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The  purpose  of  this  thesis  was  to  formulate  a  Cost 
Benefit  Analysis  method  to  be  used  when  deciding  which 
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or  program.  Due  to  shrinking  Department  of  Defense  budgets, 
Prograun  Managers  required  a  standardized  format  to  evaluate 
costs  and  benefits  of  the  three  Technical  Order  formats: 
paper,  computerized  and  automated.  This  analysis  is  general 
in  nature;  therefore,  it  can  be  applied  to  a  large  variety 
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Abstract 

This  research  focused  on  developing  a  Cost  Benefit 
Analysis  process  for  Program  Managers  to  implement  when 
deciding  what  i-3chnical  order  format — paper,  computerized  or 
automated — to  purchase.  Factors  which  determined  the 
outcome  are  costs  (life  cycle  cost  considering  technical 
order  format) ;  qualitative  issues  (characteristics  which  are 
benefits  of  a  technical  order  format) ;  and  system 
responsiveness  (time  required  to  develop  or  modify  a 
technical  order  in  each  format) . 

A  literature  review  revealed  problems  with  the  paper 
technical  order  format,  management  infrastructure, 
quantitative  cost  estimates  for  different  formats,  and 
different  techniques  which  could  be  used  for  a  cost  benefit 
analysis.  Structured  interviews  were  used  to  gather,  from 
functional  experts,  subjective  data  and  historical  data 
about  technical  orders .  The  data  collected  was  then  used  in 
one  of  three  measurement  techniques:  1)  a  simulation  model 
to  estimate  the  amount  of  time  required  to  develop  and 
modify  each  technical  order  format;  2)  a  life  cycle  cost 
analysis  spreadsheet  to  evaluate  the  cost  of  each  format;  or 
3)  an  analytical  hierarchy  to  determine  which  technical 
order  fomat  is  the  best.  On  the  basis  of  the  estimations 
and  assumptions  made  for  this  analysis,  the  research 
determined  that  the  automated  technical  order  format  is  the 
best  format . 
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REVIEW  OF  FACTORS  IMPACTING  COST  BENEFIT  ANALYSIS  (CBA) 

FOR  IMPLEMENTING 

IMPROVEMENTS  TO  THE  AIR  FORCE  TECHNICAL  ORDER  SYSTEM 

I .  Introduction 

Definition  of  Problem 

The  shrinking  Department  of  Defense  (DOD)  budget  has 
caused  DOD  acquisition  and  logistics  communities  to  look  for 
innovative  ideas  to  make  weapon  system  acquisition  and  life 
cycle  support  more  cost  efficient.  Technical  Orders  (TO) 
are  a  major  portion  of  the  total  life  cycle  cost  of  a  weapon 
system.  Today's  weapon  systems  are  becoming  increasingly 
more  difficult  to  buy  and  support  because  of  leading  edge 
technology.  As  a  result,  the  quantities  and  costs  of 
technical  orders  are  increasing  at  an  alarming  rate.  DOD 
needed  a  strategy  to  control  the  changes  to  the  acquisition 
and  logistic  support  processes .  Accordingly,  Continuous 
Acquisition  and  Life-cycle  Support  (CALS)  is  a  DOD  and 
industry  strategy  to  streamline  the  processes  for  weapon 
system  acquisition  and  logistical  support. 

CALS  is  a  strategy  to  transition  from  paper  intensive 
acquisition  and  logistic  processes  to  a  highly  automated  and 
intecrrated  mode  of  operation  for  the  weapon  systems  of  the 
1990s.  It  focuses  on  the  generation,  management, 
maintenance,  distribution  and  use  of,  as  well  as  access  to, 
technical  orders .  CALS  will  eliminate  the  development  of 
duplicate  data  and  will  also  facilitate  data  integration. 
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exchange,  and  access  among  government  and  industry 
maintained  databases . 

DOD  Instruction  5000.2  requires  that  all  Program 
Managers  perform  a  cost  benefit  analysis  to  determine  if 
conversion  to  CALS  initiatives  is  cost  effective  for  their 
program.  However,  over  the  past  eight  years,  many  Air  Force 
agencies  have  implemented  CALS  standards  on  their  technical 
order  systems,  without  first  determining  if  implementation 
was  cost  effective.  The  Air  Force  CALS  office  and  System 
Program  Offices  (SPOs)  are  interested  in  determining  how 
cost  efficient  Technical  Order  (TO)  improvements  have  been 
since  initial  implementation  of  CALS  initiatives  in  1985. 

Purpose  of  Research 

The  purpose  of  this  research  was  to  develop  a  cost- 
benefit  analysis  tool  for  decision  maker's  to  use  when 
choosing  among  alternative  TO  formats:  paper,  computerized, 
and  automated.  Air  Force  Program  Managers  (PMs)  could 
utilize  this  tool  to  determine  if  computerization  or 
automation  of  their  weapon  system  TO's,  in  accordance  with 
CALS  initiatives,  is  cost  effective.  To  accomplish  this, 
research  was  conducted  to  answer  the  following  investigative 
questions  about  Continuous  Acquisition  Life-cycle  Support 
(CALS)  and  Technical  Orders  (TOs) : 

1)  What  life-cycle  costs,  quantitative  and 

qualitative,  are  associated  with  the 
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computerization  and  automation  of  the  Air  Force  TO 
process? 

2)  What  are  the  benefits,  quantitative  and 
qualitative,  associated  with  the  computerization 
and  automation  of  the  Air  Force  TO  process? 

3)  What  CBA  approach  is  most  appropriate  for 
assessing  the  economic  consequences  of 
computerization  and  automation  of  the  Air  Force  TO 
process? 


Research  Techniques 

The  research  employed  several  different  techniques  to 
address  the  investigative  questions.  First  a  comprehensive 
literature  review  was  conducted  to  set  a  baseline  for  this 
thesis  effort.  Once  a  baseline  was  established,  the 
literature  review  and  interviews  of  TO  functional  experts 
were  used  to  answer  investigative  questions  1  and  2 .  A 
combination  of  simulation  modeling,  life  cycle  cost 
analysis,  interviews,  and  Expert  Choice  software  was  used  to 
answer  investigative  question  3.  This  research  is  limited 
to  analyzing  cost-benefit  analysis  methodologies  for  the  Air 
Force  Technical  Orders,  which  are  a  part  of  the  total 
acquisition  and  logistic  support  systems  affected  by  CALS.. 

For  this  research,  'Paper  TOs'  are  defined  as  a 
technical  order  provided  and  used  in  its  original  paper 
format.  A  'Computerized  TO'  is  simply  a  paper  technical 
order  which  has  been  formatted  into  computer  files.  An 
'Automated  TO'  is  a  technical  order  which  has  been  placed 
into  a  database  which  can  be  manipulated.  These  terms  will 
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be  used  throughout  this  paper  and  will  be  key  in  defining 
the  findings  of  our  research. 

Background 

About  the  Acronym  '  CALS ' .  The  acronym  ' CALS '  entered 
the  lexicon  of  the  Department  of  Defense  (DOD)  in  the  form 
of  a  recommendation  by  a  1984  task  force  that  studied  ways 
to  improve  logistics  support  processes  for  weapon  systems. 

At  its  introduction,  'CALS'  was  an  acronym  for  'Computer- 
Aided  Logistics  Support.'  However,  efforts  to  implement  the 
CALS  strategies  over  time  revealed  that  the  term  'Computer- 
Aided  Logistics  Support'  was  much  too  limited  to  encompass 
the  changes  contemplated  by  the  1984  task  force. 
Consequently,  in  1988,  'Computer  Aided  Logistic  Support'  was 
replaced  with  'Computer-aided  Acquisition  and  Logistics 
Support'.  In  1993,  Computer-aided  Acquisition  and  Logistic 
Support  was  again  replaced  by  'Continuous  Acquisition  and 
Life-cycle  Support '  as  the  meaning  of  the  acronym  ' CALS ' . 

The  changes  reflected  a  new  emphasis  on  continuous  process 
improvement  and  life  cycle  support.  For  clarity,,  this 
thesis  will  use  'CALS*'  as  the  acronym  for  'Computer  Aided 
Logistics  Support'  ,'CALS**'  as  the  acronym  for  'Computer- 
aided  Acquisition  and  Logistic  Support',  and  'CALS'  as  the 
acronym  for  'Continuous  Acquisition  and  Life-cycle  Support'. 

In  June  1985,  the  Joint  Industry/DOD  Task  Force  issued 
a  five  volume  report  (IDA  R-285)  which  presented  the 
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strategy  to  transition  weapon  systems  support  processes  from 
a  paper  to  an  automated  environment.  This  report,  a  direct 
result  of  the  growing  complexity  of  modern  weapon  system 
support,  initiated  the  CALS*  program.  CALS*  required  new 
weapon  system  technical  orders  (TOs) ,  engineering  drawings, 
and  other  forms  of  acquisition  data  to  be  developed,  by 
contractors,  in  a  computerized  format  and  delivered  to  DOD 
buying  activities.  The  intent  was  to  save  both  time  and 
money  by  making  use  of  modern  computer  technology  to 
alleviate  the  burden  of  supporting  paper  processes . 

In  an  effort  to  be  the  first  to  implement  DOD  policy, 
each  branch  of  the  service  implemented  its  own  version  of 
the  DOD  guidance,  resulting  in  significantly  different  and 
incompatible  systems.  On  5  August  1988,  the  Deputy 
Secretary  of  Defense  issued  the  MEMORANDUM  FOR  SECRETARIES 
OF  THE  MILITARY  DEPARTMENTS  AND  DIRECTOR,  DEFENSE  LOGISTICS 
AGENCY,  Subject:  Computer-aided  Acquisition  and  Logistics 
Support  (CALS) .  This  memorandum  directed  the  use  of  CALS** 
standards  to  ensure  compatibility  within  DOD  components  and 
contractor  data  bases.  Then,  in  1991,  the  DOD  created  a 
joint  military  program  office  to  implement  CALS**  throughout 
the  DOD  by  creating  policy,  specifying  a  standard 
architecture,  and  managing  and  controlling  resources .  This 
joint  program  office  was  designated  the  Joint  Computer-aided 
Acquisition  and  Logistic  Support  (JCALS)  program.  In 
August  1993,  the  CALS**  program  name  was  changed  to 
'Continuous  Acquisition  and  Life-cycle  Support  (CALS) . ' 
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The  history  of  the  CALS  program  indicates  that  computer 
technology  is  in  a  very  fluid  environment.  Leading  edge 
technology  is  very  expensive  and  can  become  outdated  .and  of 
little  value  in  a  short  period  of  time.  CALS  has  changed 
numerous  times  since  its  beginning  in  1985.  The  propensity 
for  change  must  be  considered  when  designing  new  automated 
computer  systems;  these  new  systems  must  incorporate 
flexibility.  Because  of  their  universal  applications,  CALS 
standards  are  great  ideas  which  have  the  potential  to 
revolutionize  the  military  acquisition  and  logistics  system. 
The  next  section  discusses  the  changes  in  the  system  and 
system  name. 

Technical  Order  Management  Syst^.  This  section 
provides  an  overview  of  recent  implementation  of  CALS 
strategies  to  the  Air  Force  Technical  Order  System  (AFTOS) . 
It  also  includes  a  discussion  on  AFTOS,  Air  Force  Technical 
Order  Management  System  (AFTOMS) ,  Joint  Uniformed  Services 
Technical  Information  System  ( JUSTIS) ,  and  Joint  Continuous 
Acquisition  and  Life-cycle  Support  (JCALS) . 

AFTOS  was  the  Air  Force's  TO  management  system  until 
the  late  1980 's,  but  it  was  criticized  for  being  highly 
ineffective  and  inefficient.  The  system  was  designed  in  the 
1940 's  and  it  was  insufficiently  updated  to  make  use  of 
current  computer  technology,  and  was  also  paper  intensive 
throughout  the  entire  life  cycle  of  a  system.  Under  AFTOS, 
TO  management  was  divided,  according  to  weapon  system,  among 
six  different  centers .  The  increasing  complexity  of  weapon 
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systems  and  the  growing  number  of  TOs  have  put  a  strain  on  ' 
the  processes  requ.ired  for  system  support.  CALS  is  a  DOD 
and  industry  strategy  developed  to  automate  and  streamline 
acquisition  and  life  cycle  support. 

To  implement  the  CALS  strategy,  in  1988  the  Air  Force 
proposed  the  Air  Force  Technical  Order  Management  System 
(AFTOMS)  which  streamlined  TO  management.  AFTOMS  was 
replaced  by  the  JUSTIS  program  in  1990;  subsequently,  JUSTIS 
was  replaced  with  the  JCALS  program  in  1991.  JCALS  retained 
most  of  the  functional  management  structure  established  by 
AFTOMS.  Its  main  emphasis  is  on  the  standardization  of  the 
TO  management  policy  throughout  each  of  the  DOD  branches . 

JCALS  is  the  streamlined  TO  management  system  of  the 
1990 's.  It  changed  the  management  structure  from  a  system 
in  which  six  different  centers  managed  different  parts  of  a 
weapon  system  to  a  four-tier  hierarchy.  The  top  of  this 
hierarchy.  The  Central  Technical  Order  Agency  (CTOA) ,  is 
responsible  for  overall  management  of  the  Air  Force  TO 
system.  The  second  tier.  Regional  Centers  (RC) ,  is 
responsible  for  acquiring  and  distributing  specific  TOs. 

Each  RC  comprises  a  number  of  Weapon  System  and  Component 
Tech  Order  Centers  (TOCs) .  The  component  TOCs  will 
eliminate  the  duplication  of  effort  by  each  weapon  system 
managing  its  own  TOs  for  components  (1:2-5) .  The  third  tier 
are  data  centers  or  libraries  located  at  each  operating 
base.  The  fourth  tier,  Work  Areas,  will  receive  TOs  in 
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digital  form  via  base  libraries  (See  Figure  1  for  functional 
flow  of  information) . 


Figure  1;  JCALS  Functional  Flow  of  Information  (1:2-6) 

To  accomplish  its  mission,  JCALS  utilizes  paper  and 
digital  TO  products  (Types  A,  B  and  C  explained  in  Chapter 
II)  .  Under  JCALS,  paper  TOs  are  to  be  'converted  to  digital 
format  whenever  it  is  economically  advantageous  to  do  so. 

To  aide  in  conversion,  JCALS  will  incorporate  Integrated 
Electronic  Technical  Manuals  (lETMs)  as  the  technology 
develops .  lETMs  depend  largely  on  new  technologies  and 
standards  such  as  Standardized  General  Mark-Up  Language 
(SGML)  . 

Ray  Technology.  Standardized  General  Markup  Language 
(SGML)  is  the  international  standard  being  developed  for 
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CALS  implementation  of  lETMs .  SGML  utilizes  tags  to  mark 
blocks  of  text  that  make  up  a  mini-database.  To  utilize 
this  powerful  new  technology,  technical  writers  need  to 
change  the  way  they  write.  Instead  of  concentrating  on 
format,  writers  must  tag  information  or  procedures  for  a 
specific  subject  so  it  can  be  used  many  times  without 
rewriting  the  procedure  or  information. 

Software  based  on  the  SGML  standard  has  been  developed 
for  inputting,  as  well  as  converting,  information  into  SGML 
format.  The  two  key  international  SOML  standards.  Document 
Style  Semantics  and  Specification  Language  (DSSSL)  and 
Standard  Page  Description  Language  (SPDL) ,  are  currently  in 
draft  and  will  be  completed  in  the  near  future.  JCALS 
stands  ready  to  incorporate  these  standards  when  they  are 
completed . 

Thesis  Overview 

Chapter  II,  Literature  Review,  presents  the  history  of 
CALS  and  the  Air  Force  TO  Management  System,  and  discusses 
capital  budgeting  and  cost-benefit  analysis  (CBA)  processes 
and  how  these  processes  can  be  applied  to  an  Air  Force  TO 
format  decision. 

Chapter  III,  Methodology,  contains  a  description  of  the 
analysis  techniques  used  in  this  study.  Structured 
interviews  were  conducted  with  functional  experts  from 
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System  Program  Offices  (SPOs)  and  Air  Logistic  Centers 
(ALCs)  . 

Chapter  IV,  Analysis  and  Findings,  is  the  analysis  of 
the  literature  review  and  structured  interviews .  A 
structured  interview  was  developed  to  determine  benefits  or 
"qualitative  issues"  for  each  TO  format.  A  computer 
simulation  model  was  developed  from  the  information  obtained 
during  the  literature  review  and  structured  interviews . 
Simulation  modeling  was  used  to  analyze  the  TO  "system 
responsiveness"  of  each  TO  format.  A  spread  sheet  was 
developed  to  provide  "cost"  data  for  each  TO  format. 

Appendix  8  lists  the  assvimptions  made  to  develop  the  Life 
Cycle  Cost  Analysis  (LCCA) .  Expert  Choice  software,  which 
utilizes  Saaty's  Analytical  Hierarchy  Process  (AHP),  was 
used  to  evaluate  quantitative  and  qualitative  costs  and 
benefits .  The  three  criteria  evaluated  for  this  CBA  were 
qualitative  issues,  cost,  and  system  responsiveness. 

Chapter  V,  Conclusions  and  Recommendations,  contains 
conclusions  drawn  from  the  data  and  recommendations  based  on 
the  analysis  results.  In  addition.  Chapter  V  presents 
recommendations  for  future  research. 


II .  Literature  Review 


Introduction 

This  chapter  1)  summarizes  the  Continuous  Acquisition 
Life-cycle  Support  (CALS)  history;  2)  compares  the  processes 
for  developing  paper  and  digital  Technical  Orders;  3) 
describes  the  structure  of  and  recent  changes  to  the  Air 
Force  TO  system;  4)  presents  a  key  international  standard 
required  for  lETMs;  5)  analyzes  the  capital  budgeting 
process;  and  6)  presents  various  accepted  approaches  to 
performing  Cost  Benefit  Analyses  (CBA) . 

CALS  History 

In  1984,  the  Institute  for  Defense  Analysis  conducted  a 
study  of  new  computer  technology  applications  to  improve  the 
logistics  support  process.  This  task  force  made  the 
following  recommendations:  1)  change  from  a  paper  intensive 
to  an  automated  logistical  and  technical  information  system; 
and  2)  acquire  and  distribute  logistics  and  technical 
information  in  computerized  format  for  new  weapon  systems 
(2:4).  In  September  of  1985,  the  Deputy  Secretary  of 
Defense  issued  a  memorandum  which  approved  the  transition 
from  paper  to  automated  weapon  system  support  processes . 

This  memorandtjm  initiated  the  Computer-Aided  Logistics 
Support  (CALS*)  program. 
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In  August  1988,  the  Deputy  Secretary  of  Defense  issued 
another  memorandum  which  directed  the  use  of  CALS** 
standards.  According  to  DODI  5000.2: 

Technical  data  that  are  required  as  deliverables, 
including  technical  manuals,  engineering  data,  and 
logistics  support  analysis  data,  should  be  required 
to  be  prepared  and  delivered  in  digital  fom  unless 
clear  convincing  analysis  shows  this  not  to  be 
cost-effective  when  assessed  across  the  life-cycle. 
(3:6-N-3) 

The  analysis  is  required  if  the  acquisition  officials  wish 
to  deviate  from  the  new  policy  of  obtaining  digital 
acquisition  and  logistics  support  data.  Therefore,  it  is 
possible,  but  not  advisable,  for  an  organization  to 
implement  CALS  without  conducting  an  analysis . 

The  CALS*  program  name  was  changed  to  Computer-aided 
Acquisition  and  Logistic  Support  (CALS**)  to  emphasize 
acquisition  involvement.  DOD  created  a  joint  office, 
composed  of  the  Defense  Logistics  Agency  and  each  branch  of 
the  military,  to  standardize  CALS  initiatives.  This  joint 
office  was  named  Joint  Computer-aided  Acquisition  and 
Logistics  Support  (JCALS**) .  The  Army  was  appointed  as  the 
lead  service  to  organize  a  single  approach  to  standardizing 
computer  hardware  and  software  for  DOD  purposes . 

In  August  of  1993,  the  CALS**  program  name  was  again 
changed  to  Continuous  Acquisition  and  Life-cycle  Support 
(CALS)  to  reflect  the  new  emphasis  on  continuous  improvement 
as  advocated  by  Total  Quality  Management  (TQM) .  The  word 
logistics  was  deleted  because  it  was  a  term  closely  related 
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to  the  military  support  system.  CALS  is  now  an  industry 
wide  movement  involving  more  than  just  the  military 
community.  The  term  acquisition,  within  the  CALS 
definition,  refers  to  smart  ways  of  acquiring  data  so  it  can 
be  used  efficiently  (4:44)  .  In  addition  to  the  CALS  name 
change,  JCALS  was  updated  to  Joint  Continuous  Acquisition 
Life-cycle  Support;  this  mirrored  the  CALS  change. 

The  government  has  taken  the  lead  to  develop  and 
organize  CALS  standards  for  industry.  The  computer 
environment  is  so  fluid  that  the  government  must  take  the 
lead  and  assume  the  financial  risks  that  are  prevalent  in 
the  early  stages  of  implementation.  These  risks  are  offset 
by  the  enomous  amount  of  savings  the  government  may  realize 
in  the  future. 

The  CALS  standards  have  grown  to  encompass  the  entire 
acquisition  and  logistic  processes  affecting  all  DOD  and 
government  contracts .  The  Defense  Department  has  committed 
more  than  $5.2  billion  to  current  or  planned  CALS  projects 
(5:50).  Although  CALS  was  originally  intended  to  improve 
military  logistics  systems,  it  has  grown  into  a  national,  as 
well  as  international,  movement  to  interchange  data  (6:24). 
This  movement,  a  key  factor  in  the  digital  revolution,  has  a 
potential  impact  equivalent  to  the  Agricultural  or 
Industrial  Revolutions  (7:12).  CALS  has  developed  into  a 
paradigm.  CALS  standards  are  a  set  of  rules  that  do  two 
things :  they  establish  boundaries  and  they  direct  behavior 
inside  those  boundaries  (8:15). 
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Paper  Versus  Digital 

These  two  words  represent  opposite  ends  of  the  spectrum 
when  it  comes  to  acquisition  and  logistic  support  systems . 

As  weapons  become  increasingly  more  sophisticated,  the 
cjuantity  and  price  per  page  for  technical  orders  and 
engineering  drawings  increase.  Simply  transforming  this 
information  into  computerized  form  will  reduce  the  volumes 
of  paper,  but  it  does  not  solve  the  problems  of  the  TO 
system  (9:17).  The  current  TO  system  is  slow,  inefficient, 
and  costly  due  to  duplication  of  effort.  For  instance. 
Logistics  Support  Analysis  (LSA)  is  used  in  the  acquisition 
process  to  design  and  evaluate  a  new  weapon  system.  LSA  is 
also  used  to  produce  technical  manuals,  but  the  LSA  record 
must  be  rewritten  for  use  as  TO’s.  The  process  duplicates 
effort  because  the  LSA  was  not  designed  to  efficiently  feed 
information  into  TO's.  The  solution  involves  gearing  the 
LSA  to  a  process  of  building  a  database  of  technical  manuals 
by  automation  (10:4)  .  In  other  words,  write  once  and  use 
many  times . 

During  the  transition  period  from  paper  to  digital 
manuals,  there  exists  three  types  of  technical  manuals. 

Type  A  manuals  are  paper  based  manuals,  and  Type  B  manuals 
are  digital  representations,  with  page  orientation,  of  the 
paper  manuals .  A  Type  C  manual  is  an  Interactive  Electronic 
Technical  Manual  (lETM)  that  utilizes  data  base  search 
techniques  to  search  the  digital  manuals .  Both  Type  B  and 
Type  C  manuals  use  the  SGML  standard  for  tagging  documents . 
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However,  Type  C  manuals  are  significantly  different  from 
Type  A  or  B.  They  have  no  page  orientation  and  the  user  must 
rely  entirely  upon  data  base  search  techniques  (11:42), 

Deficiencies  of  Paper  TOs 

The  Institute  for  Defense  Analysis  highlighted  numerous 
problems  with  paper  technical  orders.  As  was  stated  above, 
as  weapons  become  increasingly  more  sophisticated,  the 
quantity  and  price  per  page  for  technical  orders  increase. 
For  example,  a  C-5,  which  was  purchased  in  the  1960 's, 
required  600,000  pages  of  support  documents  costing  $95  per 
page,  while  the  B-IB,  purchased  in  1980 's,  required 
1, 200, 000.  pages  of  documents  at  a  cost  of  nearly  $1,  000  per 
page  (l:D-2). 

The  services  manage  hundreds  of  thousands  of  documents 
and  millions  of  pages  of  information  which  require  over  20 
million  cubic  feet  of  storage  space  to  house.  Additionally, 
more  than  $10  million  per  year  is  spent  on  postage  and 
mailing  costs  (12:4).  The  USS  Vincennes  carries  23.5  tons 
of  paper  above  the  main  deck,  which  is  more  tonnage  of  paper 
than  weapons  (11:21).  The  current  paper  system  is  very 
expensive.  The  cost  for  one  page  can  be  as  high  as  $2,000 
due  to  the  initial  research,  engineering,  editing  and 
writing  costs  associated  with  a  new  weapon  system(ll :23) . 

The  paper  system  also  produces  inaccuracies  because  of 
pressures  to  field  weapon  systems  on  schedule  and  under 
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budget.  Today's  weapon  systems  contain  leading  edge 
technology,  a  fact  which  also  increases  the  workload  on 
technical  writers . 

System  Deficiencies.  Numerous  deficiencies  are 
inherent  to  the  paper  TO  system:  slowness,  costliness,  and 
inaccurateness.  The  TO  process  is  slow,  partly  because 
contractors  frequently  fail  to  provide  installation  level 
TOs  in  time  for  Air  Force  verification.  Nearly  500  days  may 
be  required  to  fully  implement  a  routine  change  in  some 
cases . 

TOs  are  expensive  to  develop,  costing  an  average  of 
$1,000  per  page.  In  addition,  the  fact  that  the  Air  Force 
does  not  separate  the  cost  of  TO  preparation  from  the  cost 
*of  a  weapon  system  makes  cost  benefit  analysis  difficult 
(1:1-6) . 

The  TO  system  is  fraught  with  inaccuracies .  Changes 
may  arrive  with  pages  missing,  pages  out  of  sequence,  and/or 
pages  that  are  unreadable.  The  wrong  TO  or  wrong  quantities 
of  TO's  are  often  received  by  a  program  office.  Sometimes 
users  do  not  establish  accurate  requirements  for  TO's.  The 
tendency  exists  to  conduct  desk  top  analysis  instead  of 
actual  performance  of  tasks  for  TO  validation.  Increasing 
use  of  off  the  shelf  contractor  furnished  manuals,  which  do 
not  conform  to  military  standards,  confuse  maintenance  and 
operations  personnel  and  complicate  the  system  even  further. 
All  these  sources  of  inaccuracies  influence  the  operation 
and  maintenance  of  USAF  weapon  systems.  From  1977-1986,  88 
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mishaps  (damages  exceeding  $85  million)  indicated  inaccurate 
technical  data  was  a  contributing  factor  (1:1-6). 

The  Air  Force  TO  management  system  is  20  years  old, 
slow,  costly  and  inaccurate.  The  system  is  being  converted 
to  digital  data,  but  paper  will  exist  for  many  years  to 
come.  The  key  is  to  utilize  digital  data  processes  whenever 
it  is  economically  advantageous . 

Required  Solutions .  Improvements  are  required  to 
shorten  distribution  times,  maximize  the  accuracy,  and 
enhance  the  overall  efficiency  and  effectiveness  of  the 
system.  To  meet  these  objectives  hardware  and  software  were 
updated  with  the  latest  computer  technology  to  create  an 
automated  infrastructure  to  handle  the  flow  of  TOs  and 
information  throughout  the  system.  AFTOMS  was  the  first 
plan  to  automate  the  TO  system;  it  was  proposed  in  1988 
based  on  a  DOD  Task  Force  recommendation  to  implement  CALS 
standards  within  the  TO  system.  In  1991,  JCALS  replaced 
AFTOMS  and  added  emphasis  to  establish  common 
specifications,  standards  and  management  structures 
throughout  all  DOD  agencies . 

The  Air  Force  Technical  Order  System  (AFTOS) 

Purpose  of  TOs.  The  purpose  of  the  Air  Force  TO  system 
is  to  provide  Air  Force  personnel  written  guidance  for 
operating,  maintaining,  inspecting,  servicing,  supplying. 
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and  modifying  weapons  system  components.  Additionally,  TOs 
are  used  for  training  purposes  <13:3-1)  . 

History  of  TOs .  The  current  Air  Force. TO  system  is  an 
extremely  large,  paper  oriented,  logistic  support  system 
whose  operating  procedures  were  defined  in  the  1940 's.  This 
system  provides  the  official  medium  for  distributing 
technical  information,  safety  procedures,  and  instructions 
pertaining  to  Air  Force  weapon  systems .  The  original 
functional  structure  of  AFTOS  is  illustrated  in  Figure  2 
(13:1-4).  Currently,  there  are  well  over  200,000  active  TOs 
and  five  million  new/revised  pages  per  year  managed  by  six 
different  centers  and  divided  by  specific  weapon  systems 
(11:24).  A  TO  is  approximately  100-150  pages  in  length  and 
is  approximately  60  percent  text  and  40  percent  graphics. 

The  total  TO  data  base  is  approximately  20  million  pages 
(1:1-2)  . 

General  Description.  Air  Force  Regulation  8-2 
established  AFTOS  as  a  medium  for  disseminating  technical 
information,  instructions,  and  safety  records  pertaining  to 
Air  Force  systems .  Eleven  documents  establish  the  TO  system 
and  assign  responsibilities  related  to  the  acquisition  and 
management  of  TOs .  The  three  basic  steps  to  the  TO  process 
are  create,  deploy  and  manage. 

Create  TOs.  The  create  process  consists  of  planning, 
developing,  and  reviewing.  The  first  function,  planning. 
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begins  with  the  approval  of  a  Mission  Need  Statement  (MNS) 
and  an  Operational  Requirements  Document  (ORD) .  Figure  2 
references  a  Statement  of  Need  (SON)  which  has  been  replaced 
by  the  MNS  and  ORD.  An  effective  Technical  Order  Management 
Agency  (TOMA)  must  be  established  within  the  System  Program 
Office  as  early  as  possible.  A  TOMA  may  consist  of  one 
person  or  a  group  of  people;  either  way,  TOMA  is  the 
reference  name.  The  TOMA,  with  the  support  of  the  prime  Air 
Logistic  Center  (ALC) ,  develops  a  Technical  Manual 
Management  Plan  (TMMP) .  This  plan  provides  guidance  for  all 
TOs  produced  to  support  a  major  acquisition  program.  The 
inputs  to  the  planning  function  are  the  MNS,  the  Integrated 
Logistic  Support  Plan  (ILSP),  and  current  technology  for 
hardware  and  software.  The  output  is  the  TMMP. 

The  second  function,  development,  represents  the 
activities  necessary  to  produce  a  draft  version  of  the  TO 
requirements.  The  TOMA  ensures  the  contractor's  Technical 
Manual  Publication  Plan  (TMPP)  is  compatible  with  the  TMMP. 
The  TMPP  is  developed  by  the  contractor  and  is  legally 
binding;  therefore,  the  contractor  is  responsible  for  all 
problems  that  develop  during  the  contract  period.  Inputs  to 
the  development  function  are  the  Logistic  Support  Analysis 
Record  (LSAR)  data,  system  design  data,  TMMP,  and  applicable 
existing  TOs.  Outputs  include  the  TMPP  and  a  draft  TO. 

The  third  function,  review,  is  the  final  step  in  the 
create  process .  The  draft  TO  is  validated  by  the  contractor 
and  verified  by  the  Air  Force.  This  ensures  that  the 
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manuals  will  provide  explicit,  technically  accurate,  safe 
procedures.  The  input  to  this  process  is  the  draft  TO  and 
the  output  is  an  approved  preliminary  TO.  After  Review,  the 
draft  TOs  are  ready  for  deploying  (13:3-1) . 

Deploy  TOs.  The  approved  TOs  are  printed,  distributed, 
used,  and  modified  as  required  during  the  weapon  system's 
life  cycle.  The  first  function  of  the  deploy  process  is  the 
printing  of  TOs .  There  are  two  inputs  to  the  printing 
function;  1.)  approved  TOs  and  2.)  approved  changes, 
revisions  and/or  supplements  to  TOs .  The  outputs  are 
printed  TOs,  changes  and  supplements. 

The  next  deploy  function  is  to  distribute  the  printed 
TOs,  changes  and  supplements.  Normally,  the  Government 
Printing  Office  (GPO)  contractor  is  responsible  for 
distribution  in  accordance  with  the  instructions  and  mailing 
labels  provided  by  the  TOMA.  The  TOs  are  distributed  to 
regional  Technical  Order  Distribution  Offices  (TODOs)  and 
are,  in  turn,  distributed  to  the  appropriate  users .  Inputs 
to  this  function  include:  the  printed  TOs  and  changes  to 
TOs,  as  well  as  numerous  forms  and  labels  detailing 
distribution  locations  and  quantities .  Output  is  an 
approved  new  TO  or  a  changed  TO. 

Operators  and  maintainers  use  TOs  to  support  the 
equipment  that  supports  the  Air  Force  mission.  TOs  provide 
instructions  for  operating,  maintaining,  servicing, 
supplying,  and  modifying  weapon  systems .  The  input  to  the 
last  function,  modification,  is  a  printed  TO  that  has  been 
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in  use;  the  output  is  a  changed  TO  that  returns  to 
circulation.  The  changes  and  corrections  are  documented  on 
an  Air  Force  Technical  Order  Form  22  (AFTO  22) . 

Once  an  AFTO  22  is  initiated  the  modification  function 
begins.  TOs  can  be  modified  by  supplementation  (add  data 
within  sections  contained  in  the  basic  TO) ,  changes  (replace 
pages  in  basic  TO) ,  and  revision  (produce  a  subsequent 
edition  of  the  TO) .  Time  Compliance  Technical  Orders  (TCTO) 
may  affect  existing  TOs.  TCTOs  may  direct  a  change  to 
equipment  and  thus  require  a  change  to  TOs  for  operating  or 
maintaining  that  equipment.  When  TCTOs  impact  existing 
TOs,  updates  to  all  affected  TOs  are  issued  concurrently 
with  the  TCTO.  Inputs  to  this  process  are  AFTO  Form  22, 

AFTO  Form  27,  and  printed  TOs,  while  the  outputs  are 
approved  changes,  revisions,  supplements,  or  TCTOs  (13:3-1). 

Manage  TOs.  A  key  ingredient  to  the  success  of  the  TO 
process  is  effective  management.  Management  of  the  TOs 
occurs  over  the  entire  life  cycle  of  the  weapon  system.  Two 
sub-functions  make  up  the  management  function;  budgeting  and 
cataloging.  The  TOMA  and  TO  Distribution  Control  Authority 
(TODCA)  are  responsible  for  effective  management  and 
coordination  among  all  affected  organizations . 

The  TOMA  is  responsible  for  managing  the  acquisition  of 
TOs  and  is  also  required  to  deliver  a  final  TO  to  the 
appropriate  ALC  (13:6-1).  The  TOMA  is  responsible  for 
estimating  costs  associated  with  acquiring  the  necessary 
TOs.  Normally,  historical  information  is  used  for 
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predicting  future  costs  and  arriving  at  a  budget . 

Predicting  future  costs  for  digital  data  is  more  difficult 
because  there  is  little  or  no  historical  information  on 
digital  data  available  within  a  SPO.  During  acquisition, 
the  acquiring  agency  is  responsible  for  funding  the  budget. 
Once  the  weapon  system  is  delivered  to  the  operating 
command,  operations  and  maintenance  funds  are  used  for 
reproduction  of  existing  TOs  that  are  changed  or  revised. 

See  Figure  3  for  a  flow  diagram  of  the  TO  production  process 
(1:1-5). 


Figure  3 :  Flow  Diagram  of  TO  Production  Process 


The  appropriate  ALC  is  responsible  for  the  cataloging 
functions  associated  with  TOs.  The  ALC's  TODCA  is  primarily 
responsible  for  this  function,  which  includes  activities 
relevant  to  numbering,  indexing,  managing,  and  distributing 
TOs,  in  addition  to  improving  the  Air  Force  TO  System.  The 
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primary  ALC  reviews  all  requests  for  TO  numbers .  Oklahoma 
City  ALC  approves  the  TO  numbers  for  all  non-nuclear  system 
TOs  and  San  Antonio  ALC  approves  the  nuclear  related  system 
TOs.  The  primary  ALC  TODCA  manages  the  initial  distribution 
and  requisition  of  TOs .  The  appropriate  ALC  is  then 
responsible  for  maintaining,  printing,  and  distributing 
approved  TOs . 

Deficiency  of  AFTOS  System 

Complexity.  The  AFTOS  contains  over  200,000  TOs 
managed  by  8  different  prime  ALCs  and  6,000  TODOs  (11:24)  . 

In  fiscal  years  1983  through  1986,  there  was  an  average  of 
8,017  requests  for  new  TOs  with  a  98  percent  fill  rate.  The 
current  TO  inventory  data  base  is  approaching  20,000,000 
pages  (13:F-1)  and  there  are  over  50,000,000  pages  of 
unprinted  TOs .  Seventy  to  ninety  percent  of  all  printing  is 
accomplished  by  civilian  contractors.  Fifty  five  military 
specifications/standards  have  the  ability  to  impact  a  TO, 
and  over  350  forms  are  used  throughout  the  AFTOS  (1:1-6)  . 
Approximately  150  reports  are  generated  monthly  to  assist 
the  ALC  TODCA  in  managing  the  TO  stock  level  and  the  status 
of  publications  requiring  review  (13:F-1) . 


JCALS 

Purpose.  JCALS  streamlined  the  flow  of  technical 
information  by  introducing  automated  processes  within  the  TO 
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system.  JCALS  institutionalized  international  computer 
standards  within  all  DOD  agencies.  This  standardization 
throughout  the  DOD  will  help  eliminate  duplication  of 
effort . 

History.  In  October  1985,  an  Air  Force  Program 
Management  Directive  (PMD)  established  a  CALS*  Management 
Integration  Office  (MIO) .  The  MIO  was  responsible  for 
planning,  developing,  and  implementing  CALS*  initiatives 
within  the  Air  Force  (1:1-1). 

The  AFTOMS  plan,  now  referred  to  as  JCALS,  adopted  a 
seven-to-ten  year  period  for  implementation.  The  designers 
intended  the  new  system  to  evolve  from  the  AFTOS  by 
integrating  existing  assets  with  future  modules .  The 
current  system  operates  in  parallel  (1:2-1).  Also,  JCALS 
assximed  the  new  system  would  allow  the  Air  Force  to  accept 
and  efficiently  use  digital  TO  data  from  contractors. 

Structure.  JCALS  established  a  4-tier  hierarchy:  1) 
General  Administration  (Central  Technical  Order 
Administration  (CTOA) ) ,  2)  Acquisition  and  Production 
(Regional  Centers) ,  3)  Ordering  and  Distributing  (Base 
Libraries) ,  and  4)  Applications  (Work  Areas) .  See  Figure  1 
for  the  flow  diagram  of  information  path  flows  (1:2-6) . 

The  CTOA's  control  of  the  entire  system's  technical 
information  and  the  assignment  of  single  TO  Centers  (TOCs) 
to  each  weapon  system  are  key  organizational  features  of 
JCALS;  they  provide  accountability  and  control  over  the 
complete  life  cycle  of  a  TO.  TOCs  are  staffed  by  personnel 
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with  strong  backgrounds  in  weapon  system  technical  data. 

Base  Libraries/Data  Centers  are  staffed  by  people  highly 
skilled  in  computer  disciplines.  During  the  transition 
period  from  paper  to  digital  manuals,  there  will 
exist  three  types  of  technical  manuals — Types  A,  B  and  C — or 
similar  parallel  systems. 

JCALS  was  designed  to  be  an  integrated  system  that  can 
accommodate  all  types  (A,  B,  and  C)  of  TOs .  The  concept  was 
designed  to  accept  Type  C  TOs  when  they  become  available. 
With  JCALS,  the  acquisition  process  is  standardized.  The 
management  and  the  operations  of  the  CTOA  and  Regional 
Centers  provide  a  single  point  of  contact  for  questions  and 
eliminate  duplication  of  effort  by  each  weapon  system  office 
managing  its  own  component  TOs.  Automation  flexibility  at 
the  work  area  level  allows  Commands  to  proceed  at  their  own 
pace  by  selecting  configurations  that  best  suit  the 
individual  unit ' s  needs . 

Simply  transforming  TOs  into  digitized  form  will  reduce 
the  volume  of  paper  used  to  produce  the  final  product.  It 
will  also  save  time  and  space;  however,  it  does  not  solve 
all  the  problems  with  the  current  TO  system  (9:17) . 
Eventually,  the  digital  copy  of  the  TO  must  be  printed  on 
paper,  or  a  computer  screen  must  be  available  at  the  job 
site,  for  the  technician  to  use.  The  latter  solution  would 
be  inefficient  for  the  technician,  causing  him/her  to  have 
to  scroll  through  the  computer  screen  searching  for  the 
procedure  required. 
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This  inefficiency  is  why  Interactive  Electronic 
Technical  Manuals  (lETM)  are  needed.  lETMs  contain  all  the 
information  of  a  paper  TO  and  are  similar  to  a  small  data 
base  where  the  technician  can  search  the  menu  with  simple 
commands.  lETMs  have  no  page  orientation;  instead,  they 
contain  information  that  is  tagged  according  to  the  key 
international  standard  necessary  to  implement  lETMs  called 
Standard  Generalized  Markup  Language  (SGML) . 


Standard  General i zed  Markup  Language 

General  Information.  Until  recently,  the  technology 
was  not  available  to  support  digital  data.  Even  today, 
spftware  and  hardware  are  still  being  developed  to  integrate 
software  systems  and  make  digital  manuals  easier  to  use. 
Digital  manuals  cost  considerably  less  to  store  and  update 
than  paper  ones . 

Standard  Generalized  Markup  Language  (SGML)  is  the 
international  standard  for  marking  text  with  tags  so  the 
docTiment  can  be  retrieved  and  displayed  in  different  output 
formats.  The  idea  is  to  standardize  computer  files  much  as 
pipe  fittings  and  electrical  outlets  are  standardized 
(5:50)  .  SGML  makes  it  possible  to  use  electronic 
files/manuals  as  databases  by  using  SGML  sets  of  rules  to 
describe  components  of  documents  such  as  headings, 
paragraphs,  and  tables.  Currently,  software  that  utilizes 
these  standards  is  difficult  and  expensive  to  implement 
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(5:51).  Once  SGML  is  fully  developed;  however,  it  will  help 
make  digitized  manuals  more  user  friendly. 

SGML,  first  released  in  1986,  is  a  standard  for 
document  description  published  by  the  International 
Organization  for  Standardization  (lOS) .  It  uses  ASCII 
characters  (ASCII  is  accepted  by  virtually  all  publishing 
systems)  for  both  content  and  markup.  Since  ASCII 
characters  are  read  by  most  computers,  SGML  documents  can  be 
easily  exchanged  among  computers .  Perhaps  the  greatest 
feature  of  SGML  is  its  flexibility.  Information  does  not 
become  outdated  when  publishing  equipment  or  programs  are 
updated.  The  information  becomes  independent  from  the 
program  that  created  it  (14:211). 

Dedicated  work  continues  in  the  development  of  new 
standards  that  will  be  compatible  with  SOdL  and  that  will 
expand  its  implementation  by  making  output  more  easily 
automated.  Document  Style  Semantics  and  Specification 
Language  (DSSSL)  is  a  standard  that  defines  a  language  that 
specifies  how  to  build  an  SC^lL  document .  Standard  Page 
Description  Language  (SPDL)  is  the  standard  to  transform  the 
digital  document  into  output  or  a  formatted  page.  Both 
DSSSL  and  SPDL  are  now  draft  standards  and  very  close  to 
being  complete  (14:217).  Conversion  to  lETMs  will  take 
place  on  a  large  scale  (14:217)  once  the  standards  are 
complete . 

SGML  Implementations .  Worldwide  revenues  for 
products  related  to  SGML  will  more  than  double  by  1995, 
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according  to  Inter  Consult,  a  market-research  firm 
specializing  in  the  computer-publishing  market .  The  study 
predicts  that  steady  growth  is  due  to  the  shift  toward 
electronic  delivery  of  documentation  required  in 
manufacturing,  military,  telecommunications,  and  service 
fields.  As  part  of  the  shift.  Control  Data  systems 
announced  it  will  be  converting  its  paper  documentation 
system  to  electronic  manuals,  joining  Novell  and  Silicon 
Graphics  (15:32). 

Lockheed,  with  the  participation  of  the  Georgia 
Institute  of  Technology,  is  delivering  technical  manual  data 
for  the  C-130  Hercules  transport  to  Warner  Robins  Air 
Logistics  Center  on  a  $20  million  Air  Force  contract.  The 
manuals  are  being  composed  according  to  CALS  standards  using 
SOCL  (16:58) .  Using  SGML  techniques,  Ontario  Hydro  has 
automated  its  nuclear  power  plant  documentation  bringing  the 
existing  20,000  pages  of  documentation  on-line  (17:383). 


Estimated  Savings 

Estimated  savings  is  a  quantitative  benefit  for 
implementing  an  improvement  to  a  system.  This  section  will 
present  estimated  and  actual  savings  gained  due  to  CALS 
strategies.  The  example  used  previously  for  the  LSA  and 
technical  manuals  is  estimated  to  save  25  percent  of 
aggregate  costs .  The  reference  did  not  give  an  estimate  of 
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the  costs  associated  with  changing  to  the  improved  system 
(10:5)  . 

In  an  example  of  savings  gained,  CALS  standards  were 
applied  to  a  recent  modification  to  the  KC-10  on-board 
loader  project.  The  Development  Manufacturing  and 
Modification  Facility  at  Wright-Patterson  AFB  designed 
prototype  and  production  units  of  the  On-Board  Loader.  They 
transferred  this  design  to  Warner  Robins  Air  Logistics 
Center  for  manufacturing.  Because  the  two  bases  have 
different  Computer  Aided  Engineering  programs  Warner  Robins 
could  not  use  Wright-Patterson  AFB  software.  Thanks  to  CALS 
technology  the  two  systems  are  now  integrated.  The  result 
is  2,000  to  3,000  man-hours  of  savings  for  this  single 
project.  No  data  was  available  for  the  costs  associated 
with  integrating  these  two  systems  (18:15). 

Under  the  current  engineering  drawing  system,  the  DOD 
spends  millions  of  work-  hours  revising  engineering  data  from 
paper  and  other  forms .  The  DOD  could  save  millions  of  work 
hours  if  intelligent  digitized  data  were  distributed  on  all 
weapon  systems  (19:38) .  Once  again  the  reference  did  not 
provide  information  on  how  much  it  would  cost  to  change  the 
entire  system  over  to  an  intelligent  data  system.  The 
stories  of  savings  are  numerous;  however,  more  information 
is  required  before  deciding  to  implement  CALS  standards  on  a 
given  program.  It  would  be  a  mistake  to  implement  CALS  in 
every  situation,  unless  it  is  cost  effective.  Currently, 
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there  is  no  auditing  process  that  will  provide  verification 
and  validation  for  the  amount  of  time  and  savings  generated 
by  implementing  CALS  standards  and  Integrated  Data  Base 
management  (20:21) . 


Economic  Analysis 

Economic  analysis  is  a  four-step  process:  1) 
identification  of  possible  projects,  2)  data  collection,  3) 
determining  the  best  project,  and  4)  evaluation  of 
implemented  projects  (21:5)  .  '"A  good  economic  analysis 
systematically  examines  and  relates  costs,  benefits,  and 
risks  of  various  alternatives'”  (21:6). 

Costs  are  normally  the  easiest  to  determine,  but 
benefits  and  risks  can  be  much  more  difficult  to  estimate. 
Costs  can  be  provided  by  a  contractor's  estimate,  or 
estimated  from  historical  information.  Standard  capital 
budgeting  procedures  can  be  used  to  conduct  a  cost  analysis 
to  determine  the  best  project — the  project  with  the  lowest 
life  cycle  cost.  However,  a  good  economic  analysis  is 
normally  not  baaed  on  cost  alone 

As  stated  above,  a  good  economic  analysis  considers 
costs,  benefits,  and  risks.  The  standard  capital  budgeting 
analysis  is  not  designed  to  handle  qualitative  factors  or 
benefits  and  risks.  Capital  budgeting  can  be  used  to 
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estimate  costs  but  another  tool  is  required  to  combine 
quantitative  and  qualitative  factors.  Therefore,  capital 
budgeting  analysis  must  be  combined  with  another  decision 
making  tool  in  order  to  accomplish  a  good  CBA  or  economic 
analysis.  A  complete  economic  analysis  should  be  considered 
as  one  of  the  inputs  required  to  make  a  proper  decision 
concerning  the  use  of  resources,  and  not  as  the  decision 
making  process  itself  (22:8). 

The  next  part  of  this  chapter  describes  various  capital 
budgeting  analysis  and  decision  making  tools  that  can  be 
used  to  conduct  the  economic  analysis  portion  of  a  CBA.  The 
analysis  in  Chapter  IV  utilizes  the  net  present  value  method 
for  capital  budgeting  and  Saaty's  AHP  for  a  decision  making 
tool . 


Capital  Budgeting  Analysis 

Capital  budgeting  is  a  quantitative  Economic  Analysis 
technique  that  the  Air  Force  could  use  for  TO  system 
improvement  decisions.  Capital  budgeting  and  cost  analysis 
involves  long  term  planning  decisions  for  investments  using 
both  quantitative  and  qualitative  cost  information.  This 
process  is  tailored  for  projects,  or  for  the  life  of  a 
project,  not  a  specific  time  period.  The  process  aims  at 
minimizing  life  cycle  costs,  costs  incurred  over  the  life  of 
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the  system.  Because  this  is  a  long  range  plan  most  of  the 
costs  should  be  variable  and  not  fixed. 

There  are  four  steps  to  the  capital  budgeting  and  cost 
analysis  process : 

1.  Analyze  quantitative  financial  aspects  of  a 
project. 

2 .  Analyze  qualitative  non-financial  aspects  of  a 
project. 

3.  Finance  the  project. 

4 .  Implementing  the  project  &  monitoring  its 
performance.  (23:673) 

Only  relevant  costs,  future  costs  that  differ  among 
alternatives,  should  be  used  in  the  analysis. 

Four  Capital  Budgeting  Factors.  A  manager  must 
consider  four  capital  budgeting  factors:  Quantitative/ 
financial,  quantitative/non-financial,  qualitative/ 
financial,  and  qualitative/non-financial .  The  cjuantitative 
/financial  factors  c?.n  be  analyzed  by  one  of  the  four 
methods  outlined  below.  Sometimes  assumptions  and 
predictions  can  be  made  to  obtain  a  quantitative /monetary 
cost  or  savings,  and  sometimes  a  manager  must  make  decisions 
based  on  all  four  categories  of  factors . 

Step  One.  The  most  difficult  aspects  of  capital 
budgeting  are  identifying  the  project  and  predicting  the 
outcomes.  Simulation  modeling  can  be  used  to  improve  the 
accuracy  of  predicting  outcomes  of  a  complicated  decision. 
When  a  model  is  used,  the  inputs  to  the  model  are  predicted 
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and  the  outcome  of  the  entire  system  is  provided  by  the 
model.  Also,  modeling  can  help  quantify  benefits; 
the  results  are  easier  to  use  in  capital  budgeting  and 
economic  analysis.  Analyzing  the  quantitative  and  financial 
aspects  of  a  project  may  be  accomplished  using  one  of  four 
different  methods:  Discounted  Cash  Flow  (DCF)  Method, 

Payback  Method,  Accrual  Accounting  Rate  of  Return  (ROR) ,  and 
Break-Even  Time  (BET)  Method  (23:674). 

Discounted  Cash  Flow.  When  comparing 
alternatives,  DCF  considers  the  time  value  of  money.  A 
dollar  today  is  not  worth  a  dollar  one  year  from  now  because 
of  inflation  and  return  on  investments  demanded  by  profit 
seeking  companies .  DCF  uses  a  dollar  value  or  interest  rate 
of  return  to  compare  costs  and  benefits  of  alternatives . 
There  are  two  types  of  DCF:  net  present  value  and  internal 
rate  of  return. 

The  Net  Present  Value  (NPV)  method  gives  current  dollar 
value  of  cost  and  savings  in  today's  dollars.  If  inflation 
is  ten  percent,  or  a  company  requires  a  ten  percent  return 
on  the  investment,  then  a  100  dollar  payment  1  year  from  now 
is  presently  valued  at  approximately  $91. 

Internal  Rate  of  Return  (IRR)  provides  an  interest  rate 
of  return  at  which  expected  outflows  equals  expected 
inflows .  The  higher  the  value  of  the  IRR  the  better  the 
investment.  An  IRR  imparts  the  same  information  as  a  NPV 
but  it  is  presented  in  a  different  form.  For  example,  if 
the  expected  inflow  this  year  equaled  91  dollars  and  the 
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expected  outflow  one  year  from  now  equaled  100  dollars,  an 
IRR  of  ten  percent  would  be  required. 

Payback.  The  Payback  Method  is  another  system  for 
analyzing  quantitative  financial  aspects  of  a  program.  This 
method  provides  information  on  the  amount  of  time  the 
required  to  recoup  investment  costs.  It  is  computed  by 
dividing  total  investment  outflow  costs  by  the  yearly 
savings  from  the  investment.  This  number  is  straight 
forward,  easy  to  compute,  and  used  when  a  quick,  approximate 
solution  is  required.  The  objective  is  to  minimize  the 
payback  time.  This  method  does  not  account  for  the  time 
value  of  money,  or  the  life  cycle  profitability  of  a 
project.  For  example  one  project  may  have  a  smaller  payback 
time  but  less  of  a  life  cycle  than  an  alternative  project. 

If  the  decision  was  based  on  payback  time  alone  the  decision 
would  be  erroneous . 

Accrual  Accounting  Rate  of  Return.  The  next 
method  is  the  Accrual  Accounting  ROR.  This  method  utilizes 
traditional  financial  accounting  methods,  which  measure 
profitability,  and  ignores  the  time  value  of  money.  It  is 
calculated  as  an  accounting  measure  of  income  divided  by 
investment.  The  higher  the  ROR  the  better  the  project. 

This  method  is  not  as  accurate  as  DCF  because  it  utilizes 
accounting  averaging  techniques  which  ignore  the  time  value 
of  money.  Use  of  this  method  will  not  always  result  in  the 
selection  of  the  best  alternative.  However,  managers  that 
use  DCF  methods  will  be  criticized  in  the  short  run  when 
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executives  compare  income  statements  with  results  from  NPV 
or  IRR  methods . 

Break-Even  Time.  The  last  method  is  the  Break- 
Even  Time  (BET) .  This  is  defined  as  the  time  from  the  start 
of  a  project,  date  of  idea,  to  when  the  cumulative  present 
value  of  the  cash  inflow  equals  the  cash  outflow.  The 
objective  is  to  choose  the  shortest  possible  alternative 
BET.  This  method  is  extremely  important  in  manufacturing 
industries  where  fierce  competition  means  the  first  item 
available  for  sale  captures  the  market. 

Which  Method  is  Right?  There  is  not  one  method 
that  is  best  for  all  situations .  Each  method  presented 
above  is  appropriate  depending  on  the  situation.  Each 
manager  must  decide  which  is  the  most  appropriate  for  the 
decision  at  hand.  One  way  to  make  this  decision  would  be  to 
list  the  advantages  and  disadvantages  of  each  method  for  the 
particular  situation.  For  example,  a  manager  must  decide  if 
electronic  technical  manuals  are  more  cost  effective  than 
paper  technical  orders.  To  do  this  he/she  must  determine 
which  cost  analysis  method  to  use.  The  NPV  method  would  be 
ideal  for  the  TO  scenario  because  of  the  accuracy  gained  by 
using  DCF  and  the  large  amount  of  initial  costs  required  to 
implement  changes . 

Complex  real  world  problems  are  not  as  easy  to  solve 
as  text  book  problems .  The  factors  that  managers  must 
consider  are  far  broader  than  the  quantitative  and  financial 
factors  required  in  capital  budgeting  decisions.  Many 
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factors  are  difficult  to  quantify.  These  qualitative  and 
non-financial  factors  can  be  as  important  as,  or  sometimes 
even  more  important  than,  the  quantitative  and  financial 
factors . 

Step  Two.  The  four  methods  presented  above  are 
designed  to  give  quantitative  information  for  decision 
makers.  The  next  step  in  the  Capital  Budgeting  process  is 
to  analyze  qualitative  non-financial  aspects  of  the  project. 
Managers  must  also  consider  both  types  of  information.  Each 
manager  must  decide  how  to  weigh  each  type  Of  information 
before  making  the  final  decision.  A  manager  may  wish  to 
obtain  quantitative  information  using  a  capital  budgeting 
method  and  then  combine  this  with  the  qualitative  factors 
affecting  the  decision.  He/she  can  then  prioritize  the 
factors.  In  other  words,  the  manager  could  list  the 
advantages  and  disadvantages,  qualitative  and  quantitative, 
for  each  project. 

Both  qualitative  and  quantitative  financial  information 
are  analyzed  by  decision  makers  before  implementing  capital 
budgeting  investments,  and  both  have  limitations. 

Qualitative  and  quantitative  information  may  be  based  on 
predictions  of  perceived  costs  and  benefits,  which  are  not 
always  100  percent  accurate.  Estimates  are  least  accurate 
when  dealing  with  leading  edge  technology,  because  there  is 
little  or  no  historical  information  on  the  subject . 

Even  quantitative  information  can  be  highly  subjective. 
For  example,  in  order  to  obtain  information  on  cost  and 
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savings  for  implementing  a  capital  investment,  an  estimate 
of  costs  and  benefits,  in  monetary  terms,  must  be  made. 
Normally,  these  estimates  can  be  obtained  from  historical 
information  or  collected  from  experts  using  a  questionnaire 
or  survey.  Both  quantities  are  estimates  which  are  subject 
to  bias  and  error,  because  all  scenarios  are  different.  If 
the  item  being  considered  for  the  capital  investment  is 
similar  to  another  investment  that  was  implemented  in  the 
past,  then  the  information  should  be  accurate  enough  for 
predicting  cost  and  benefits.  If  there  are  no  similar  items 
to  use  for  comparison,  then  the  predictions  are  much  more 
difficult . 

How  can  qualitative  and  quantitative  information  be 
combined  to  make  the  decision?  Qualitative  information  can 
be  subjective,  perhaps  even  more  so  than  qualitative 
predictions.  It  is  difficult  to  integrate  qualitative  with 
quantitative  information  in  order  to  make  a  capital 
budgeting  decision.  The  integration  process  can  be  biased 
and  inaccurate;  however,  the  analysis  can  be  improved  by 
using  a  sensitivity  analysis  (changing  the  factors  and 
observing  the  outcome)  and  comparing  predictions  with 
historical  information. 

Step  Three.  The  third  step  in  capital  budgeting  is  to 
finance  the  project.  Once  a  decision  is  made  on  which 
project  will  be  implemented  it  is  time  to  gather  and  commit 
funds .  In  the  military  environment  funding  is  becoming 
harder  to  accomplish  because  of  shrinking  budgets .  In  a 
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military  situation,  the  budgeting  process  takes  place  many 
years  in  advance  of  project  implementation. 

Step  Four.  The  final  step  is  to  follow  through  with 
implementation  and  to  monitor  the  performance.  The  project 
may  get  out  of  control  if  it  is  not  closely  monitored.  The 
decision  to  implement  the  project  will  be  based  on 
predictions;  therefore,  the  information  gained  from 
monitoring  the  process  could  be  very  helpful  in  predicting 
future  costs  and  savings . 

Decision  Kaking  Tools 

CBA  decisions  can  be  based  on  a  single  quantitative 
criterion  such  as  NPV  cost.  In  such  a  case,  the  choice  of 
an  optimal  solution  is  not  difficult;  rating  alternatives  is 
only  a  matter  of  mathematically  calculating  the  total  cost 
or  value  to  the  organization.  Once  each  solution  is  rated, 
the  decision  maker  chooses  the  alternative  with  the  lowest 
cost  or  greatest  value  (21:10-11). 

Most  decision-making  methods  may  work  well  for  a  single 
criterion  problem,  but  they  may  not  be  the  most  effective 
approach  for  a  problem  with  several  issues,  such  as  the 
transfer  to  a  new  TO  format.  Because  CALS  requires  several 
different  issues,  a  Multiple  Criteria  Decision  Making  (MCDM) 
should  be  implemented.  There  are  eight  common  approaches  to 
MCDM:  1)  Single  Objective,  2)  Goal  Progrcunming,  3) 
Interactive,  4) Compromise  Programming  5)  Electre,  6) 
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Parametric,  7)  De  Novo  Programming,  and  8)  Analytical 
Hierarchy  (21:13-16). 

The  first  seven  techniques  are  quantitative  in  nature. 
The  decisions  that  result  from  using  these  techniques  are 
commonly  based  on  measurable  characteristics,  but  they  do 
not  take  into  account  the  qualitative  issues  which  are 
important  to  this  study.  "A  MCDM  is  needed  that  can  combine 
quantitative  and  qualitative  factors  into  the  . . .  decision" 
(24:37).  Therefore,  each  of  the  first  seven  MCDMs  can  be 
used  as  a  tool  to  aid  the  decision  maker  in  performing  a 
CBA,  but  a  more  comprehensive  approach  must  be  used  to  do 
the  overall  CBA. 

Saaty'8  Analytic  Hierarchy  Process  (AHP) 

The  eighth  process,  AHP  "is  a  multilevel  decision  aid 
developed  by  Thomas  Saaty  in  the  1970's"  (21:17).  It  is  one 
of  the  most  popular  aids  to  decision  making  developed  in  the 
past  decade  (25:57). 

The  Analytical  Hierarchy  Process  is  a  rational  and 
systematic  approach  fop  finding  a  solution  to  a  problem. 

The  method  allows  decision  makers  to  partition  large, 
manageable  problems  into  smaller  parts  that  are  easier  to 
handle.  It  provides  decision  makers  with  the  ability  to 
combine  qualitative  and  quantitative  criteria  to  form  a 
rating  for  each  alternative  solution.  These  ratings  may 
then  be  used  as  a  basis  for  project  selection  (24  :37)  . 
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The  AHP  method  establishes  ratings  and  rankings  for 
each  solution  choice  by  using  a  system  of  pairwise 
comparisons.  The  ratings  are  then  mathematically  combined 
to  develop  an  overall  rating  and  ranking  for  each  of  the 
alternatives.  The  alternatives  can  be  compared,  regardless 
of  their  units,  and  an  educated  decision  can  be  made.  This 
method  also  provides  a  process  for  checking  the  consistency 
among  the  importance  of  the  decision-maker's  preferences  of 
the  multiple  criteria  (21:16). 


The  use  of  AHP  includes  the  following  steps : 


1.  Build  a  decision  hierarchy  by  breaking  the  general 
problem  into  several  issues,  which  are  positioned 
within  a  hierarchical  structure. 

2.  Gather  relational  data  for  the  decision  criteria. 
Relational  data  may  exist  in  the  form  of  quantitative 
relationships,  or  it  may  be  generated  through  a  series 
of  pairwise  comparisons  of  the  decision  criteria  and 
alternatives . 

3.  Estimate  the  relative  weights  of  the  decision 
criteria  using  the  proportional  or  the  eigenvalue 
method  (uses  data  to  assign  weights) . 

4 .  Aggregate  the  relative  weights  into  a  vector  that 
will  be  used  to  rank  the  various  decision  solutions . 
(21:18) 

Battin  and  Bender  detail  these  steps  completely  in  their 
thesis  about  economic  analysis  (21:18-28) . 

Selecting  Multiple  Alternatives  of  AHP.  After 
establishing  the  ratings,  the  decision  maker  is  provided 
with  the  best  solution  or  the  highest  rated  alternative. 
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Unfortunately,  management  may  be  able  to  select  more  than 
one  alternative,  or  small  differences  in  requirements 
between  different  decision  .situations  may  cause  a  need  for  a 
different  solution.  A  combination  of  alternatives  may  be 
required.  One  possible  solution  to  this  problem  is  to  use 
integer  linear  programming  to  choose  the  combination  of 
alternatives  that  would  give  the  largest  combined  rating. 

The  values  in  the  solution  would  indicate  how  many  times 
each  alternative  should  be  implemented.  Zero-one 
programming  should  be  used  to  insure  that  solutions  used 
only  once  will  not  be  given  too  much  weight .  Zero-one 
programming  is  a  subset  of  linear  integer  programming  that 
restricts  the  values  of  possible  solutions  to  either  a  zero 
or  a  one.  The  result  is  that  a  solution  will  either  be 
chosen  or  not  chosen.  In  this  environment  it  is  an 
important  requirement  that  the  ratings  be  additive. 
Therefore,  the  sum  of  one  combination  of  ratings  will  be 
directly  comparable  to  another  (21:29) . 

Consistency  of  AHP.  Because  decision  makers  can  be 
inconsistent,  Saaty  developed  a  technique  for  assessing  the 
consistency  of  their  judgments .  The  decision  maker  analyzes 
the  maximum  eigenvalue  and  the  number  of  criteria  in  a 
pairwise  comparison.  Although  this  sounds  difficult,  there 
is  now  a  computer  program.  Expert  Choice,  which  accomplishes 
the  analysis  for  the  decision  maker.  Battin  and  Bender 
detail  this  method  for  us  in  their  thesis.  This  measurement 
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allows  decision  makers  to  be  confident  in  their  solution 
choice . 

Criticism  of  the  AHP.  In  many  applications  of  the  AHP, 
two  hierarchies,  and  hence,  two  output  vectors,  are 
associated  with  a  given  situation,  one  for  its  'benefits' 
and  the  other  for  its  'costs'.  Here  the  question  arises  as 
to  how  these  two  vectors  should  be  combined  to  decide  which 
alternative  to  implement  (24:57). 

Bernhard  and  Canada  stated  that  ratios  of  data  do  not 
provide  the  decision  maker  with  the  "best"  solution  (21:42/ 
25:57),  but  their  conclusion  has  been  successfully  rebutted 
by  two  counter  arguments.  First,  it  is  inappropriate  to 
separate  items  into  costs  and  benefits  if  they  can  be 
measured  in  common  units .  Second,  the  integrity  of  their 
formula  depends  on  the  commonalty  of  the  units  involved. 

Conclusion 

The  history  of  the  CALS  program  indicates  that  computer 
technology  environment  is  very  fluid.  Leading  edge 
technology,  which  is  very  expensive,  can  become  outdated  and 
of  little  value  in  a  short  period  of  time.  CALS  has  changed 
numerous  times  since  its  beginning  in  1985.  The  propensity 
for  change  must  be  considered  when  designing  new  automated 
computer  systems;  these  new  systems  must  incorporate 
flexibility.  Because  of  their  widespread  applicability. 
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CALS  standards  are  great  ideas  which  have  the  potential  to 
revolutionize  the  military  acquisition  and  logistics  system. 

Joint  Continuous  Acquisition  and  Life-cycle  Support  has 
streamlined  the  management  of  TOs  within  the  Air  Force.  The 
system's  emphasis  on  process  improvement  is  the  key  to 
incorporating  state-of-the-art  computer  standards  into  the 
TO  system  as  they  become  available. 

Under  AFTOS,  the  Air  Force  did  not  efficiently  and 
effectively  manage  the  increasing  size  and  complexity  of 
Weapon  System  TOs .  The  manually-oriented,  paper  processes 
and  fragmented  management  structure  caused  the  system  to  be 
inaccurate,  slow,  and  costly. 

JCALS  utilizes  an  improved  management  structure  and 
implements  the  use  of  new  computer  technology.  The  system 
has  eliminated  the  burdensome  paper  processes  for  review  and 
distribution  of  TOs,  It's  success  can  be  contributed  to  the 
flexibility  of  its  implementation.  The  system  is  designed 
for  digital  TOs  to  evolve  over  a  7-10  year  period  during 
which  a  parallel  structure  of  paper  and  digital  TOs 
harmoniously  exist  together. 

lETMs  are  expected  to  be  the  next  generation  of  TOs . 
These  lETMs  utilize  SGML  software  standards,  which  are 
nearly  complete.  Development  of  lETMs  will  continue  to 
thrive  throughout  all  of  DOD  and  industry.  With  JCALS  in 
place  the  Air  Force  is  positioned  to  take  advantage  of  the 
latest  in  technology. 
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A  great  deal  of  literature  is  available  that  provides 
information  about  estimated  savings  from  converting  to 
digital  TOs,  but  very  little  literature  exists  for  costs.  A 
good  analysis  must  be  based  on  both  costs  and  benefits . 
Several  cost  benefit  analysis  methods  have  been  presented, 
along  with  the  advantages  and  disadvantages  of  each. 

Capital  Budgeting  is  ideal  for  quantitative/monetary 
analysis.  The  NPV  is  an  accurate  measurement  of  life  cycle 
costs  for  a  large  capital  investment,  such  as  changing  TO 
format .  Capital  budgeting  does  not  incorporate  qualitative 
data  effectively  into  the  analysis;  therefore,  when  using 
only  capitol  budgeting,  the  decision  maker  is  forced  to 
consider  two  separate  categories  of  infoimiation :  qualitative 
and  quantitative.  A  better  method  would  be  to  implement 
Saaty's  AHP . 

Saaty's  AHP  combines  the  quantitative  and  qualitative 
information  needed  and  provides  one  simple  recommendation  to 
the  decision  maker.  Implementing  improvements  to  the  Air 
Force  TO  format  requires  evaluation  of  both  qualitative  and 
quantitative  data.  Therefore,  the  Analytical  Hierarchy 
Process  is  an  ideal  approach  to  developing  a  CBA  methodology 
for  this  application. 

The  literature  review  has  provided  an  overview  of  CALS, 
JCALS,  capital  budgeting,  and  CBA  techniques.  Chapter  III 
will  present  the  methodology  used  to  answer  the 
investigative  questions  outlined  in  Chapter  I . 
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Ill .  Httthodology 


latroduction 

This  chapter  describes  the  methodology  used  to  answer 
the  investigative  questions  proposed  in  Chapter  I.  The 
purpose  of  this  research  was  to  identify  a  procedure  for  a 
SPO  to  use  when  conducting  a  Cost  Benefit  Analysis  (CBA) 
before  making  a  decision  on  whether  or  not  to  acquire 
digitized  technical  data.  The  literature  review  provided 
historical  information  about  many  areas  including  CALS  and 
JCALS.  In  addition,  various  cost  benefit  analysis 
techniques  were  analyzed.  Structured  interviews  were  then 
conducted  to  gather  data  from  functional  experts .  The 
functional  experts '  data  provided  inputs  for  determining 
benefits,  building  a  computer  simulation  model,  conducting  a 
Life  Cycle  Cost  Analysis,  and  performing  a  cost  benefit 
analysis  using  Saaty's  Analytic  Hierarchy  Process. 

Justification  for  Method  Selected 

Because  CALS  is  a  relatively  new  initiative,  no 
guidance  exists  on  how  program  managers  would  accomplish  a 
CBA  to  assess  the  cost  effectiveness  of  implementing  CALS 
strategies  on  weapon  system  support  processes .  In  these 
days  of  shrinking  budgets,  it  is  important  to  maximize  the 
value  of  a  dollar;  therefore,  without  a  good  CBA  it  is 
impossible  to  know  if  CALS  implementation  on  a  given  program 
is  cost  effective. 
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Measurement  data  is  important  for  management  to  justify 
continuation  of  a  program.  This  research  effort  needed 
measurement  data  to  determine  the  best  CBA  solution.  In 
order  to  obtain  this  measurement  data,  a  two  fold  approach 
was  necessary:  literature  review  and  personal  interviews. 
Research  began  by  studying  cost-benefit  analysis  techniques. 
After  looking  at  various  techniques,  information  and 
estimations  were  gathered  to  determine  which  technique  was 
most  appropriate  for  the  situation.  Measurement  data  was 
obtained  from  interviews  and  the  literature  review.  The 
data  was  then  transformed  into  decision  making  information 
by  developing  the  LCCA,  simulation  model,  and  then 
conducting  the  CBA  using  Expert  Choice  software. 

Methodology 

To  answer  investigative  questions  1  and  2,  structured 
interviews  with  members  from  System  Program  Offices  (SPOs) 
and  Air  Logistic  Centers  (ALCs)  were  conducted.  The 
interviews  identified  categorical  costs  and  benefits  of 
implementing  the  CALS  initiatives.  The  interviewees  also 
ranked,  on  a  Likert  scale,  the  benefits  of  alternative  TO 
formats:  paper,  computerized,  and  automated.  Forty-three 
interviews  were  conducted  in  SPOs,  ALCs,  and  Matrix 
organizations .  The  number  of  interviews  to  be  conducted  was 
determined  by  the  CALS  office  and  the  thesis  committee.  The 
interviews  helped  identify  costs  incurred  and  benefits 
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received  for  each  TO  format.  The  interview  responses  were 
analyzed  to  identify  the  qualitative  and  quantitative  costs 
and  benefits  that  are  contmon  and  those  that  were 
organizationally  dependent.  A  computer  simulation  model  was 
developed  to  simulate  the  process  the  Air  Force  utilizes  to 
develop  and  modify  TOs.  The  model  was  used  to  analyze  the 
system  responsiveness — time  to  develop  or  modify  a  TO. 
Interview  and  literature  review  data  were  input  into  the 
simulation  model.  Sensitivity  analysis  was  used  to 
determine  the  effect  of  input  changes . 

Questions  1  and  2  established  the  essential  benchmarks 
to  determine  costs  and  benefits  for  all  three  TO  formats . 

The  information  collected  from  answering  investigative 
questions  1  and  2  were  input  into  Expert  Choice  software, 
which  utilizes  AHP .  The  "best"  CBA  method  was  defined  as 
the  method  which  has  the  capability  to  analyze  quantitative 
and  qualitative  information.  The  AHP  has  this  capability 
and;  therefore,  provides  a  format  for  decision  makers  to  use 
in  developing  a  CBA  prior  to  implementing  costly 
improvements .  This  aids  in  ensuring  that  resources  are 
expended  wisely.  The  CBA,  which  utilizes  the  costs  and 
benefits  identified  in  questions  1  and  2,  is  presented  in 
Chapter  IV. 

Question  3  was  answered  by  conducting  an  extensive 
literature  review  on  CBA  techniques,  as  well  as  structured 
interviews  with  functional  experts  from  SPOs  and  ALCs . 
Methods  for  conducting  cost  benefit  analyses  were 
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researched.  Each  method  was  analyzed  by  comparing  the  type 
of  costs  and  benefits,  to  decide  which  method  was  the  most 
appropriate.  This  information  was  obtained  from  the 
interviews  and  literature  review.  "Most  appropriate"  is 
defined  as  the  CBA  method  which  can  analyze  quantitative  and 
qualitative  costs  and  benefits.  The  sources  for  CBA  methods 
consisted  of  theses,  pamphlets,  books,  and  regulations. 


Summary 

This  chapter  described  the  methodology  used  to  answer 
the  investigative  questions.  A  combination  of  literature 
review  and  personal  interviews  were  used  to  answer  all  the 
investigative  questions .  The  next  chapter  will  describe  the 
interview,  simulation  model,  cost  analysis  and  the  CBA 


method . 


IV.  Analysis  and  Findings 


Introduc±ion 

The  purpose  of  this  research  was  to  answer  the 
following  investigative  questions  about  Continuous 
Acquisition  Life-cycle  Support  (CALS)  and  Technical  Orders 
(TOs)  : 

1)  What  life  cycle  costs,  quantitative  and 
qualitative,  are  associated  with  the 
computerization  and  automation  of  the  Air  Force  TO 
process? 

2)  What  are  the  benefits,  quantitative  and 
qualitative,  associated  with  the  computerization 
and  automation  of  the  Air  Force  TO  process? 

3)  What  CBA  approach  is  most  appropriate  for 
assessing  the  economic  consequences  of 
computerization  and  automation  of  the  Air  Force  TO 
process? 


In  order  to  answer  these  questions,  several  different 
processes  had  to  be  utilized.  This  chapter  details  1)  the 
interview  questions  and  format  used;  2)  the  computer 
simulation  models  used;  3)  the  Life  Cycle  Cost  Analysis 
(LCCA)  developed;  and  4)  the  Analytical  Hierarchy  Process 
(AHP)  output  developed  from  the  research.  The  answers  to 
the  investigative  questions  will  be  presented  in  Chapter  V. 


Interviews 

Formulation.  A  structured  interview  was  developed  to 
gather  information  about  CALS  and  TOs.  The  objective  of 
this  interview  was  to  determine  which  characteristics  are 
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benefits  of  different  TO  formats  and  what  were  the 
historical  costs  of  these  formats  (See  Appendices  3  &  4) . 
After  formulating  our  interview,  we  compared  its  contents  to 
a  C-17  TO  Business  Case  Analysis  conducted  by  the  CALS 
Shared  Research  Center  (CSRC) ,  San  Antonio  (See  Appendix  5). 
We  found  that  most  of  the  issues  determined  to  be  important 
to  our  study  were  also  questioned  in  their  study.  This 
comparison  validated  our  research  by  exhibiting  that  the 
issues  discussed  are  important  to  System  Program  Offices 
(SPOs)  . 

After  formatting  the  interview,  we  called  various  SPOs 
to  identify  an  experienced  representative  who  was  willing  to 
be  interviewed.  We  received  forty-three  positive  responses, 
representing  four  bases:  four  SPOs  with  27  interviewees;  one 
Air  Logistics  Center  (ALC)  with  one  interviewee;  and  two 
matrix  organizations  with  fifteen  interviewees.  The 
structured  interviews  were  facsimiled  to  the  representatives 
before  the  personal  interview;  this  allowed  them  to  be  aware 
of  the  topics  of  discussion  and  to  accumulate  any  supporting 
data  that  was  needed.  After  giving  the  representatives  time 
to  review  the  structured  interviews,  we  contacted,  them  again 
to  establish  personal  interview  dates. 

The  first  step  was  to  record  which  SPO  was  being 
interviewed,  what  TO  format  the  SPO  was  currently  using  and 
the  experience  of  the  interviewee.  These  facts  helped  us 
identify  causes  of  trends  that  resulted  from  the  interviews . 
Next,  the  table  of  characteristics  and  related  formats  was 
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discussed  and  completed  (See  Appendix  3) .  The  reasoning 
behind  the  responses  was  recorded  to  identify  biases  and 
trends  of  the  interviewees .  Finally,  we  collected  the 
quantitative  historical  data  for  the  SPOs  latest  TO 
acquisition. 

Results.  The  summary  output  from  the  first  page  of  the 
interviews  is  provided  in  the  next  five  tables.  The 
interviewees  were  asked  to  rank  characteristics  of  an  ideal 
TO.  Table  1  is  a  compilation  of  characteristic  ranks  for 
the  ideal  TO.  Column  one  lists  characteristics  considered. 
The  interviewees  were  allowed  to  add  or  delete 
characteristics  from  Column  one.  Column  two  is  the  sum  of 
the  points  received  by  each  characteristic.  Column  three  is 
the  number  of  times  a  characteristic  was  ranked  on  the 
interviews .  Column  four  is  the  average  number  of  points 
received  by  each  characteristic.  The  average  was  obtained 
by  dividing  column  two  by  column  three.  Column  five  is  the 
rank  of  each  characteristic.  This  number  was  determined  by 
ranking  the  average  points  received,  lowest  average  first. 
TABLE  1:  DATA  FOR  OVERALL  RANKING  OF  CHARACTERISTICS 
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Tables  2-4  are  a  compilation  of  ranks  of  the 


characteristics  for  the  paper,  computerized  and  automated 
TO  formats,  respectively.  -The  method  used  to  calculate  the 
rank  is  basically  the  same  as  the  method  used  for  Table  1, 
except  a  Likert  scale  from  one  to  five  was  used  to  quantify 
the  interviewees  input.  A  score  of  one  meant  the 
interviewee  definitely  agreed  that  the  characteristic  was  a 
benefit  of  that  TO  format;  a  score  of  five  meant  the 
interviewee  strongly  disagreed.  The  characteristic  with  the 
lowest  average  received  the  highest  ranking. 


TABLE  2;  DATA  FOR  PAPER  FOR] 

sIAT 

PAPER 

TOTAL  PTS 

#  OF  OBS 

AVERAGE 

RANK 

MAINTAINABLE 

109 

31 

3.51 

7 

RELIABLE 

91 

26 

3.50 

6 

SECURITY 

114 

38 

3.00 

3 

ACCURACY 

110 

41 

2.68 

2 

EFFICIENCY 

115 

30 

3.83 

10 

EFFECTIVENESS 

120 

34 

3.53 

8 

TIMELY 

102 

34 

3.00 

3 

COST 

130 

32 

4.06 

11 

DEPLOYABLE 

84 

38 

2.21 

1 

FLEXIBILITY 

55 

18 

3.06 

5 

SUPPORTABILITY 

34 

9 

3.78 

9 

TABLE  3:  DATA  FOR  COMPUTERIZED  FORMAT 


COMPUTERIZED 

TOTAL  PTS 

#  OF  OBS 

AVERAGE 

RANK 

MAINTAINABLE 

34 

18 

1.88 

3 

RELIABLE 

58 

17 

3.41 

9 

SECURITY 

50 

22 

2.27 

7 

ACCURACY 

43 

22 

1.954 

5 

EFFICIENCY 

41 

21 

1.952 

4 

EFFECTIVENESS 

70 

22 

3.18 

8 

TIMELY 

55 

26 

2.12 

6 

COST 

53 

14 

3.78 

11 

DEPLOYABLE 

42 

28 

1.50 

1 

FLEXIBILITY 

34 

22 

1.54 

2 

SUPPORTABILITY 

31 

9 

3.44 

10 
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TABLE  4:  DATA  FOR  AUTOMATED  FORMAT 


AUTOMATED 

TOTAL  PTS 

#  OF  OBS 

AVERAGE 

RANK 

MAINTAINABLE 

55 

38  . 

1.45 

2 

RELIABLE 

49 

25 

1.96 

8 

SECURITY 

83 

35 

2.37 

9 

ACCURACY 

56 

36 

1.50 

4 

EFFICIENCY 

37 

31 

1.19 

1 

EFFECTIVENESS 

62 

37 

1.68 

6 

TIMELY 

53 

34 

1.56 

5 

COST 

101 

26 

3.88 

11 

DEPLOYABLE 

70 

37 

1.89 

7 

FLEXIBILITY 

29 

19 

1.53 

3 

SUPPORTABILITY 

29 

9 

3.20 

10 

Table  5  contains  the  ranks  compiled  from  the  previous 
four  tables.  This  table  allows  us  to  compare  how  different 
formats  are  viewed,  as  well  as  determine  if  there  is  a 
difference  of  views  when  each  TO  format  is  compared  to  the 
ideal  TO  format. 


TABLE  5;  OVERALL  RANKING  DATA  FROM  SURVEYS 


OVERALL  RANKS 

PAPER 

COMPUTER 

AUTO 

IDEAL 

MAINTAINABLE 

7 

3 

2 

6 

RELIABLE 

6 

9 

8 

2 

SECURITY 

3 

7 

9 

9 

ACCURACY 

2 

5 

4 

1 

EFFICIENCY 

10 

4 

1 

3 

EFFECTIVENESS 

8 

8 

6 

4 

TIMELY 

3 

6 

'  5 

5 

COST 

11 

11 

11 

10 

DEPLOYABLE 

1 

1 

7 

7 

FLEXIBILITY 

5 

2 

3 

11 

SUPPORTABILITY 

9 

10 

10 

8 

When  the  interviewees  ranked  the  characteristics, 
without  regard  to  format.  Accuracy  was  ranked  as  a  definite 
first.  As  you  can  see  in  Table  5,  accuracy  was  never  ranked 
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first  for  an  individual  format.  Some  respondents  decided  to 
only  rank  the  top  five  characteristics  for  a  format,  this 
meant  the  remaining  characteristics  were  not  considered 

benefits  for  a  particular  TO  format.  The  data  from  the 

\ 

individual  format  rankings  was  used  to  make  the  pairwise 
comparisons  in  the  AHP . 

As  stated  in  Chapter  II,  zero-one  programming  should  be 
used  to  ensure  that  solutions  used  only  once  will  not  be 
given  too  much  weight.  Therefore,  three  responses  provided 
by  participants  were  not  considered:  price,  usability  and 
readability.  The  research  team  also  decided  that  any 
characteristic  receiving  less  than  thirty  observations  in 
two  or  more  categories  should  be  removed  from  consideration. 
Therefore,  flexibility,  supportability,  cost,  and 
reliability  were  not  considered.  These  two  decisions 
reduced  the  number  of  qualitative  characteristics  used  for 
the  AHP  from  eleven  to  seven . 

The  data  collected  from  page  two  of  the  interview  was 
designed  to  provide  information  for  the  LCCA;  however,  the 
interview  data  was  heavily  slanted  towards  paper  TOs.  This 
slant  was  the  result  of  the  interviewees  having  limited 
experience  with  computerized  and/or  automated  TO  formats,  as 
well  as  limited  experience  with  cost  information. 
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Background.  According  to  Pritsker,  models  are 
descriptions  of  systems .  Models  are  usually  developed  based 
on  theoretical  laws  and  principles;  they  may  be  scaled 
physical  objects,  mathematical  equations  and  relations,  or 
graphical  representations  (27:5).  The  research  we  conducted 
included  the  use  of  three  different  models,  one  for  each  TO 
format.  The  models  predict  the  time  required  to  develop  and 
modify  a  TO. 

To  develop  our  models  we  used  SLAM  II.  The  state-of- 
the-art  simulation  language,  SLAM,  is  the  first  commercial 
language  that  provided  three  different  viewpoints  in  a 
single  integrated  format.  It  permits  discrete  event, 
continuous,  and  network  modeling  perspectives  and/or 
combinations  of  the  three  (28:1)  .  A  network  model  is  a 
graphical  picture  of  a  system.  This  approach  decomposes  a 
complex  problem  into  its  elements  (27:7).  Information  can 
be  obtained  directly  from  the  network  diagram  and  it 
provides  a  basis  for  a  qualitative  or  quantitative 
assessment  and  understanding  of  the  mechanisms  that  are 
under  the  control  of  the  decision  makers  (27:8-9) .  The  SLAM 
II  sturanary  output  report,  found  in  i^pendix  6,  displays  the 
statistical  results  for  the  simulation  (27:98). 

Our  models  (See  Appendix  6)  depict  the  steps  that  a  TO 
follows .  Due  to  differences  in  aunount  of  time  required  to 
complete  a  task,  a  different  model  was  developed  for  each  TO 
format.  Assumptions  for  each  model  had  to  be  made,  see 
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Appendix  6.  We  will  present  the  model  descriptions,  discuss 
the  differences  between  the  three  models,  and  present  the 
output  from  the  simulation. 

Description.  The  model  is  a  series  of  nodes  (events 
occurring  which  change  the  state  of  the  TO)  and  activities 
(time  durations).  See  J^pendix  5.  In  the  models,  a  TO 
begins  at  the  creation  stage;  8000  TOs  are  created  every 
year.  It  then  enters  a  development  stage  with  a  time 
duration  following  the  stage-  The  time  duration  depicts  the 
amount  of  time  required  to  develop  the  TO.  After 
development,  it  enters  a  print  stage,  and  a  distribution 
stage.  Each  of  these  stages  also  has  a  time  duration 
associated  with  it.  After  distribution,  the  amount  of  time 
required  to  develop  or  modify  a  TO  is  collected. 

Development  time  is  collected  for  new  TOs;  modification  time 
is  collected  for  TOs  already  in  use. 

Once  modification  or  development  time  is  collected,  the 
TO  then  becomes  part  of  the  functional  TO  system.  There  are 
approximately  200,000  Technical  Orders  presently  in  the 
system.  All  of  these  TOs  are  considered  to  be  in  use,  and  a 
percentage  of  the  TOs  in  use  are  modified  each  year.  Those 
that  are  modified  are  sent  to  the  error  node,  where  the 
model  simulates  modifications  to  the  TO.  The  TO  is  re¬ 
distributed  and  returns  to  use  until  another  modification  is 
required.  All  three  models  start  with  the  200,000  TOs  in 
the  USAF  inventory.  We  know  that  approximately  eight 
thousand  TOs  are  created  annually;  therefore,  one  thousand 
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are  created  every  46  days.  If  a  TO  requires  a  modification, 
it  would  begin  at  the  fix  stage  and  then  follow  the  same 
remaining  order  of  stages  outlined  above. 

Differences.  It  follows  that  the  three  TO  formats 
would  hot  require  the  same  amount  of  time  to  complete  the 
stages  listed  above.  These  models  contain  rough  order 
magnitude  (ROM)  estimates  for  all  time  distributions .  Time 
distributions  for  the  develop,  print,  distribute,  and  fix 
activities  we  e  e '-timated  by  the  research  team,  based  on  the 
literature  review  and  interviews,  and  then  input  into  the 
appropriate  model.  The  estimates  and  assumptions  are  listed 
in  i^pendix  7 . 

Development.  The  paper  and  computerized  formats 
require  the  same  amount  of  time  to  develop,  because  they 
both  utilize  the  same  process .  The  main  difference  between 
the  two  formats  is  the  output .  The  paper  format  is 
delivered  as  paper  TOs;  whereas,  the  computerized  formats 
are  delivered  as  a  computer  file.  The  automated  format  can 
be  developed  more  rapidly  than  the  other  two  formats  because 
of  efficiencies  obtained  from  using  high  technology 
processes.  Once  automation  is  embraced  on  a  large  scale 
throughout  the  DOD  acquisition  process,  there  will  be  a 
greater  savings  in  time  due  to  the  learning  curve. 

Print.  From  interviews  with  experienced  TOMAs,  it 
was  determined  that  paper  technical  orders  require  an 
average  of  ninety  days  to  be  printed;  however,  print  time 
may  vary  from  as  little  as  ten  days  to  as  many  as  365  days. 
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Computerized  TOs  are  printed  at  the  base  library  after 
electronic  distribution.  The  printing  time  at  the  library 
requires  an  average  of  20  days.  Automated  TOs  are  not 
printed;  therefore,  no  time  is  associated  with  this 
activity.  Distribution  of  a  paper  TO  requires  approximately 
fifteen  days;  however,  automated  and  computerized  formats 
are  distributed  over  telephone  lines;  therefore, 
distribution  requires  only  one  day. 

Modification.  According  to  literature,  eight 
percent  of  all  paper  TOs  are  modified  each  year. 

Modification  of  these  TOs  usually  requires  fifty  days .  In 
contrast,  computerized  TOs  need  modification  six  percent  of 
the  time,  requiring  fifty  days  on  average,  and  automated  TOs 
need  modification  three  percent  of  the  time,  requiring 
forty-five  days  on  average.  All  of  the  above  differences 
were  incorporated  into  their  respective  models;  they  caused 
significant  variations  in  the  output. 

Output.  The  following  table  is  a  summary  of  the 
desired  data  obtained  from  running  the  three  SLAM  II  models. 
The  first  column  lists  the  format  considered,  the  second 
column  lists  the  nximber  of  days  required  to  modify  a  TO,  and 
the  third  column  lists  the  number  of  days  required  to 
develop  a  TO. 
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TABLE  6:  SLAM  II  OUTPUT 


FORMAT 

MODIFICATION 

DEVELOPMENT 

PAPER 

230 

574 

COMPUTERIZED 

75 

444 

AUTOMATED 

47 

356 

TOTAL  DAYS 

352 

1374 

After  obtaining  this  data,  we  added  the  total  time  to  modify 
and  divided  each  format  time  by  the  total  time.  We  then 
found  the  reciprocal  of  these  percentages.  After  finding 
the  reciprocal,  we  added  the  numbers  again  and  found  the 
percentage  for  each  format.  This  percentage  was  entered 
into  the  AHP .  The  following  table  contains  the  end  product 


of  the  above  calculations: 

TABLE  7:  PERCENTAGES  ENTERED  INTO  AHP 


FORMAT 

MODIFICATION 

DEVELOPMENT 

PAPER 

0.112 

0.256 

COMPUTERIZED 

0.343 

0.331 

AUTOMATED 

0.545 

.413 

TOTAL  DAYS 

1.0 

1.0 

Life  Cycle  Costing 

Background.  Costs  can  be  categorized  according  to  the 
following  four  distinct  phases  of  a  project:  1)  Research  and 
development,  2)  Acquisition,  3)  Operations  and  Support,  and 
4)  Disposal.  Each  project  will  follow  a  chronological  path 


through  each  phase.  A  Life  Cycle  Cost  Analysis  (LCCA)  is 
normally  accomplished  at  the  system  or  program  level,  but 
can  be  beneficial  for  evaluating  subsystems  as  well.  The 
key  to  proper  use  of  the  LCCA  at  the  subsystem  level  is  to 
maintain  a  systems  perspective,  which  is  essential 
considering  the  impact  of  different  alternative  subsystems 
on  the  complete  system.  Maintaining  this  perspective 
requires  the  analyst  to  have  a  broad  view  of  the  process  or 
the  program. 

If  an  alternative  optimizes  a  subsystem,  but  not  the 
complete  system,  then  suboptimization  has  occurred,  or  the 
systems  view  has  been  violated.  A  military  TO  management 
system  is  a  subsystem  which  affects  the  LCC  of  the  entire 
weapon  system.  Appendix  10  is  our  example  of  using  LCCA 
with  a  system  view.  The  systems  view  has  no  room  for 
suboptimization  of  the  entire  system.  For  example,  the  TO 
cost  cannot  be  minimized  at  the  expense  of  the  whole  system. 

Each  phase  of  a  new  project  incurs  costs .  The  risk 
associated  with  implementing  new  technology  must  be  weighed 
along  with  the  costs  and  benefits  of  the  project. 
Accordingly,  when  considering  changes  to  the  format  of 
technical  orders,  one  must  consider  the  risks,  costs  and 
benefits.  Cost  and  savings  estimations  are  an  analyst's 
attempt  to  predict  the  future.  Historical  data  is  normally 
used  in  the  estimation  process;  this  practice  is  itself 
risky.  However,  when  no  historical  data  exists,  and  new 
technology  is  being  developed,  the  risks  are  even  greater . 
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This  research  uses  the  systems  approach  to  determine 
the  overall  effect  of  the  life  cycle  costs  of  three 
different  alternative  formats  for  technical  orders .  The 
system  approach  requires  an  analysis  of  the  LCCs  for  the 
entire  system.  The  analyst  must  determine  what  impact  a  TO 
format  has  on  the  entire  life  cycle  cost  of  a  system.  The 
weapon  system  life  cycle  costs  for  each  format  must  be 
estimated  and  compared.  Once  the  most  cost  effective  format 
is  chosen,  the  data  can  be  weighted  and  input  into  the  AHP. 

Development,  when  developing  the  LCCA  for  TOs,  we  made 
several  assumptions;  detailed  in  Appendix  8.  The  data  we 
used  to  formulate  the  equations  were  obtained  from  the 
literature  review,  interviews,  and  an  Armstrong  Laboratory 
study.  This  analysis  contains  rough  order  magnitude  (ROM) 
cost  estimates,  which  were  developed  by  the  research  team, 
when  literature  review  and  interview  data  did  not  provide 
the  data. 

Cost  estimates  for  the  automated  TO  format  were 
obtained  from  the  Armstrong  Laboratory  cost  benefit  study 
conducted  on  the  F-16  aircraft  (29) .  Our  study  converted 
the  Armstrong  Laboratory's  ROM  estimates  into  unit  costs  for 
one  aircraft  and/or  TO  page,  which  we  then  used  in  the  LCCA 
equations.  The  costs  were  divided  into  two  categories, 
non-recurring  and  recurring. 

Output.  The  formulas  used  for  computation  of  the  LCC 
and  the  output  from  LCCA  can  be  found  in  Appendices  9  &  10. 
Using  1700  aircraft,  575,000  TO  pages  and  a  period  of  ten 
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years  for  our  input;  we  computed  a  sample  LCCA  (See  Appendix 
10) .  From  this  LCCA,  we  determined  the  costs  in  the 
following  table: 


TABLE  8:  SUMMARIZED  LCCA  OUTPUT 


FORMAT 

TOTAL  COST 

MET  PRESENT  VALUE 

PAPER 

$5,986,738,500 

$4,338,823,381 

COMPUTERIZED 

$4,582,706,000 

$3,389,875,881 

AUTOMATED 

$2,096,420,700 

$1,553,697,556 

The  total  costs  were  computed  as  direct,  then  year  dollars. 
The  net  present  value  is  the  total  cost  converted  to  present 
year  dollars .  These  totals  were  weighted  and  used  as  inputs 
to  the  AHP. 

Saaty's  Analytical  Hierarchy  Process 

Overview.  As  discussed  in  Chapter  II,  AHP  is  a 
rational  and  systematic  approach  for  finding  a  solution  to  a 
problem.  It  allows  a  large  problem  to  be  broken  down  into 
smaller,  more  manageable  segments .  AHP  allows  both 
qualitative  and  quantitative  criteria  to  be  compared  at  the 
same  time,  using  a  system  of  weights.  We  used  AHP  to 
compare  the  output  from  the  interviews,  the  LCCA,  and  the 
SLAM  II  model.  To  apply  AHP  to  the  information  we  used  a 
software  program  titled  Expert  Choice  (30) .  Expert  Choice 
is  an  interactive  program  that  can  be  run  on  personal 
computers;  it  is  designed  to  apply  Saaty's  Analytic 
Hierarchy  Process  to  the  obtain  the  solution  of  problems. 
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Expert  Choice  helps  the  user  establish  a  matrix  of  pairwise 
comparisons  by  asking  a  series  of  interactive  questions  that 
systematically  develops  the  relative  importance  (weights)  of 
the  characteristics  (31:29).  An  AHP  decomposes  a  decision 
problem  into  three  stages:  1)  goal,  2)  levels  of  evaluation 
criteria,  and  3)  alternatives. 

I  Figure  4  is  the  completed  hierarchy  for  our  study.  It 

resulted  from  compiling  several  smaller  steps:  1)  creating  a 
hierarchy  using  the  information  from  the  interviews;  2) 
developing  a  hierarchy  from  the  SLAM  II  data;  3)  determining 
the  weighted  data  from  the  LCCA;  and  4)  and  combining  the 
three  inputs  into  a  final  hierarchy. 

Step  1.  The  interviews  asked  functional  TO  experts  to 
rank  eleven  characteristics  on  the  basis  of  their 
desirability  for  TOs.  Only  seven  of  the  eleven  were  found 
to  be  highly  desirable,  these  seven  were  used  in  creating  a 
hierarchy  for  the  "qualitative  issues"  (QI) .  The  goal  of 
the  qualitative  issues  hierarchy  was  to  compile  weights  for 
the  three  TO  formats  with  respect  to  the  characteristics . 
This  was  accomplished  by  placing  the  ranked  seven 
characteristics  (alternatives),  as  determined  by  the 
interviews,  under  each  TO  format  (evaluation  criteria) .  The 
characteristics  were  entered  in  the  order  that  they  were 
ranked  and  weighted  accordingly  (See  Appendix  11) .  This 
hierarchy  provided  us  with  an  overall  ranking  of  the  seven 
characteristics  and  weights  for  each  of  the  three  formats . 
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Stap  2.  A  "system  responsiveness"  hierarchy  was 
created  to  determine  which  format  was  the  most  responsive  to 
development  and  modifications  of  TOs  and  to  compile  weights 
for  each  format .  The  characteristics  were  develop  and 
modify;  the  alternatives  were  paper,  computerized,  and 
automated.  The  output  from  the  SLAM  II  model  was  stated  in 
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number  of  days  required  to  either  develop  or  modify  a 
technical  order.  We  ranked  the  three  formats  with  respect 
to  number  of  days  required  (lowest  number  receiving  the 
highest  ranking) .  Because  most  of  the  TO  life  cycle  is 
spent  in  use  or  modification,  the  research  team  determined 
that  modification  should  be  weighted  heavier  than 
development.  The  decisions  and  data  were  input  into  Expert 
Choice  and  the  resulting  hierarchy  is  in  Figure  4 . 

Step  3.  A  "cost"  hierarchy  was  created  by  simply 
ranking  the  costs  as  computed  by  the  LCCA.  The  lowest  cost 
format  was  ranked  highest.  This  ranking  produced  the 
weights  required  for  input  into  the  final  hierarchy. 

Step  4.  The  goal  of  the  final  hierarchy  (See  Figure  4) 
was  to  determine  which  TO  format  was  the  best  for  SPO 
managers  to  purchase.  To  develop  this  hierarchy  we  combined 
the  three  previous  hierarchies .  Level  one  criteria  were 
cost,  system  responsiveness  and  qualitative  issues.  Level 
two  criteria  were  the  combination  of  criteria  from  the 
system  responsiveness  and  qualitative  issues  hierarchies. 

The  alternatives  were  paper,  computerized  and  automated. 

The  weights  for  the  level  one  criteria  were  determined  by 
the  research  team.  The  numbers  were  subjectively  assigned 
using  information  from  the  literature  review  and  an 
interview  with  Mr.  Clyde  Chapman  from  ASC/AL.  Mr.  Chapman 
was  chosen  because  of  his  broad  experiences  with  AF  TOs 
(33) .  Each  decision  maker  may  have  a  different  opinion 
about  the  weights  given  to  each  criterion.  For  example,  a 
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particular  PM  may  decide  on  weights  of  0.5  for  QIs,  0.3  for 
cost  and  0.2  for  system  responsiveness,  depending  on  the 
needs  of  the  particular  system.  The  weights  for  the 
alternatives  and  level  two  criteria  were  all  calculated  in 
the  previous  hierarchies . 

Summary.  The  compilation  of  the  smaller  hierarchies 
allowed  easier  formulation  of  the  final  hierarchy.  This 
hierarchy  allowed  us  to  weigh  qualitative  and  quantitative 
issues  at  the  same  time  and  to  reach  a  justifiable  decision 
(discussed  in  Chapter  V) . 

I 

Conclusion 

This  chapter  details  the  processes  utilized  to:  1) 
collect  and  evaluate  the  information  from  the  interviews/  2) 
determine  the  amount  of  time  required  to  develop  and  modify 
a  TO  in  each  format;  3) determine  costs  associated  with 
different  formats  of  technical  orders;  and  4) develop  our 
final  hierarchy  and  determine  the  best  TO  format. 

First,  we  discussed  the  interviews  and  the  assumptions 
made  to  ensure  that  the  final  list  of  characteristics  used 
was  valid.  Next,  we  discussed  the  SLAM  II  simulation  model 
and  the  niomber  of  days  required  by  each  format  to  develop  or 
modify  a  TO.  Then,  we  discussed  the  assumptions  and 
equations  used  to  formulate  the  life  cycle  cost  analysis. 
Finally,  we  presented  the  process  used  to  formulate  the 
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hierarchy;  this  hierarchy  aids  in  determining  which  format 
is  the  best  for  a  manager  to  purchase. 

Chapter  V  answers  the  investigative  questions  and 
presents  the  results  and  conclusions  of  the  research. 
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V.  Conclusions  and  Racomniendations 
Thasis  ^Conclusions 

Today's  Program  Managers  (PMs)  are  placing  increasing 
emphasis  on  efficiency  over  the  life  cycle  of  a  weapon 
system.  The  driving  factor  behind  this  emphasis  is 
budgetary  constraints.  Life  Cycle  Costs  can  be 
significantly  reduced  by  improving  the  processes  used  to 
develop  and  operate  today's  weapon  systems.  The  cost  for 
technical  data  is  a  major  contributor  to  the  total  life 
cycle  coat  of  a  system;  however,  the  quantity  of  pages  and 
cost  per  page  for  paper  technical  orders  (TOs)  are 
increasing  at  an  alarming  rate.  Computer  technology  can  be 
used  to  improve  the  TO  system  and  thereby  lower  the  cost  for 
technical  data  and  the  life  cycle  cost  of  a  weapon  system. 

The  purpose  of  this  research  was  to  develop  a  method 
for  conducting  a  cost  benefit  analysis  prior  to  implementing 
changes  to  the  format  of  a  TO.  Towards  that  ends,  we  sought 
to  answer  the  following  three  investigative  questions : 

Investigative  Question  One.  "What  life  cycle  costs, 
quantitative  and  qualitative,  are  associated  with  the 
computerization  and  automation  of  the  Air  Force  TO  process?" 
This  question  was  answered  by  the  data  from  interviews  and 
the  life  cycle  cost  analysis. 

Each  interview  consisted  of  two  parts:  the  first  part 
gathered  information  on  the  qualitative  benefits  of  the 
ideal  TO  and  each  TO  format,  while  the  second  part 
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concentrated  on  the  quantitative  costs  of  each  format.  The 
second  part  of  the  interviews  were  used  to  answer 
investigative  question  one.  The  Life  Cycle  Cost  Analysis 
(LCCA)  used  a  systems  approach  to  estimate  the  quantitative 
life  cycle  costs .  Data  from  the  interviews  and  the 
literature  review  were  used  to  estimate  costs  for  the  LCCA. 

Response  to  the  second  part  of  the  interview  was  very 
limited.  Only  six  of  the  43  interviewees  provided  answers 
to  the  second  part,  and  the  information  given  was  limited  to 
costa  for  paper  format  only.  Costs  for  paper  TOs  varied 
from  $1,000  to  $2,500  per  page  for  a  new  TO  and  $246  to  $925 
per  page  for  changes .  The  interviewees  did  not  provide  all 
the  information  requested  due  to  lack  of  experience  with 
computerized  and  automated  TOs,  as  well  as  to  lack  of 
familiarity  with  coats .  Appendix  3  contains  a  blank 
interview  form  and  a  summary  of  responses  to  part  one  of  the 
interviews .  Appendix  4  is  a  summary  of  the  responses 
obtained  from  part  two  of  the  interviews .  Although 
Continuous  Acquisition  Life-cycle  Support  (CALS)  has  been  in 
existence  since  1985,  a  limited  amount  of  information, 
concerning  computerized  and  automated  TOs,  has  been  passed 
on  to  the  working  level  technical  order  managers  (TOMAs) . 
Also,  there  was  a  noticeable  lack  of  information  about 
costs,  as  evidenced  by  the  fact  that  only  six  out  of  43 
responses  contained  answers  to  the  second  part  of  the 
interview. 
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The  best  source  for  quantitative  costs  was  the 
Armstrong  Laboratory  (AL)  cost  benefit  study  conducted  on 
the  F-16  aircraft  in  1993  (29) .  This  study  was  used  as  a 
starting  point  for  developing  the  LCCA.  The  total  costs  in 
the  AL  study  were  divided  by  either  the  number  of  F-16s  or 
the  number  of  TO  pages  to  determine  a  per  unit  cost .  Unit 
costs  were  needed  for  developing  the  LCCA  spreadsheet,  which 
was  used  to  estimate  the  overall  life  cycle  costs  for  a 
system.  The  spreadsheet,  as  developed,  is  a  generic  tool 
which  can  be  used  by  any  TOMA  or  PM  to  help  choose  either  TO 
conversion  or  new  TO  acquisition.  The  user  would  need  to 
enter  the  number  of  aircraft,  the  life  cycle  of  the  weapon 
system,  and  the  niamber  of  TO  pages.  We  used  1700  aircraft, 
10  years  and  575,000  pages  of  TOs  for  our  LCCA. 

Costs  were  categorized  as  recurring  and  non-recurring. 
The  non-recurring  costs  are  fixed  costs  to  be  incurred 
within  the  first  year,  while  the  recurring  costs  are 
variable  costs  that  are  incurred  on  a  yearly  basis 
thereafter.  The  non-recurring  costs  are:  consulting,  data 
acquisition,  hardware,  software,  telecommunications,  and 
training  (Note:  Data  conversion  replaces  data  accpaisition, 
when  the  purpose  of  the  analysis  is  to  assess  converting 
paper  format  TOs  into  computerized  or  automated  formats) , 

The  recurring  costs  are:  TO  usage,  training,  TO 
changes,  troubleshooting,  communications,  hardware 
maintenance,  software  maintenance,  mishaps  due  to  TO  errors, 
mailing  costs,  ^.elecommunication  costs,  and  AF  TO  management 
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manpower.  Appendix  9  contains  the  formulas  that  were  used 
for  each  cost  category,  and  Appendix  10  contains  the 
numerical  answers  for  our  example.  The  spreadsheet  also 
discounts  the  life  cycle  cost  with  a  discount  rate  of  seven 
per  cent . 

The  quantitative  life  cycle  costa  for  our  scenario  are 
$4,338,823,381  for  the  paper,  $3,389,875,881  for  the 
computerized,  and  $1,553,697,556  for  automated  TO  formats. 
The  automated  format  has  the  lowest  cost  and,  hence,  is  the 
preferable  choice,  when  considering  costs.  One  must  keep  in 
mind  that  the  automated  and  computerized  cost  formulas  in 
the  LCCA  are  estimates  and  have  not  been  confirmed  by  actual 
historical  data;  therefore,  there  is  more  risk  associated 
with  automated  and  computerized  TO  formats*. 

Investigative  Question  Two.  "What  are  the  benefits, 
quantitative  and  qualitative,  associated  with  the 
computerization  and  automation  of  the  Air  Force  TO  process?" 
This  question  was  answered  by  data  from  the  literature 
review,  part  one  of  the  interviews,  and  the  simulation 
model.  Qualitative  benefits  obtained  from  the  interview  and 
quantitative  benefits  obtained  from  the  simulation  model 
were  used  to  answer  investigative  question  two. 

Part  one  of  the  interviews  asked  TOMAs  within  the 
System  Prograun  Offices  (SPOs)  and  Air  Logistics  Centers 
(ALCs)  to  identify  benefits  of  an  ideal  TO  and  of  each  TO 
format:  paper,  computerized,  and  automated.  The  responses 
are  summarized  in  Tables  1-5  of  Chapter  IV.  Based  on  the 
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responses  from  43  interviews,  the  top  seven  characteristics 
were  used  in  our  analysis  as  qualitative  benefits.  The 
remaining  characteristics  were  omitted  because  of  a  low 
average  score  received  or  insufficient  number  of  ratings  by 
the  interviewees.  All  characteristics  which  were  not  rated 
at  least  30  times  in  two  or  more  areas  were  discarded  from 
the  analysis  and  assumed  to  have  negligible  impact  on  the 
CBA.  The  qualitative  benefits  for  TOs  are:  accuracy, 
efficiency,  effectiveness,  timeliness,  maintainability, 
deployability,  and  security.  Characterist.  definitions  are 
included  on  the  interview  form.  Appendix  3 . 

In  general,  quantitative  data  is  more  commonly  used  by 
decision  makers  than  is  qualitative  data.  Therefore,  a 
simulation  model  was  developed  to  provide  a  quantifiable 
estimate  for  TO  system  responsiveness.  The  simulation  model 
estimated  system  responsiveness,  the  time  to  modify  or 
develop  a  TO,  for  the  three  different  formats:  paper, 
computerized,  and  automated.  The  model  provided  the  number 
of  days  to  develop  or  modify  a  TO  for  each  TO  format.  The 
model  ran  for  a  five  year  period,  and  an  average  value  was 
obtained  from  the  simulation  model  output  report.  Based  on 
the  assumptions  and  results  contained  in  Appendix  7,  the 
automated  foxnnat  received  the  highest  rating  for  both  TO 
development  and  modification,  requiring  an  estimated  356 
days  to  develop  and  47  days  to  modify.  The  computerized  TO 
format  required  444  days  to  develop  and  75  days  to  modify. 
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The  paper  TO  format  required  574  days  to  develop  and  230 
days  to  modify  a  TO. 

Investigative  Question  Three.  "What  CBA  or  decision 
making  technique  is  most  appropriate  when  deciding  among 
alternative  improvements  for  the  Air  Force  TO  system?"  The 
literature  review  revealed  different  methods  which  could  be 
used  for  conducting  this  analysis .  We  used  Expert  Choice  to 
apply  the  analytical  hierarchy  process  (AHP)  because  of  the 
need  to  evaluate  both  quantitative  and  qualitative  costs  and 
benefits.  The  interviews  and  the  literature  review 
established  that  cost,  system  responsiveness,  and 
qualitative  issues  (QIs)  are  three  criteria  which  determine 
the  best  TO  format.  Costs  (obtained  from  the  LCCA)  are 
quantitative;  System  responsiveness  (obtained  from  the 
simulation  model)  is  a  quantitative  benefit;  and  QIs 
(obtained  from  the  interview  process)  are  qualitative 
benefits . 

Figure  4  in  Chapter  IV  outlines  the  analytical 
hierarchy  process  used  for  our  analysis.  Based  on  the 
weights  given  to  each  of  the  criteria  and  the  costs  and 
benefits  outlined  in  investigative  questions  one  and  two, 
the  best  TO  format  is  the  automated.  The  automated  format 
received  a  rating  of  0.45,  the  computerized  format  received 
a  rating  of  0.29,  and  the  paper  format  received  a  value  of 
0.26.  According  to  the  Analytical  Hierarchy  Process  the 
alternative  with  the  largest  rating  is  the  best  alternative; 
therefs-re,  automated  is  the  best  TO  format  to  purchase. 
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RacCTimendationa 


Lessons  Learned.  We  Interviewed  functional  experts, 
concentrating  specifically  on  TO  managers  or  TOMAs.  There 
were  two  reasons  why  we  focused  on  TOMAs.  First,  we  wanted 
information  from  people  who  are  in  the  acquisition  process 
and  deal  with  TOs;  and  second,  we  were  constrained  by  time 
and  the  number  of  people  we  could  interview  in  the  time 
frame  available  for  this  research.  If  time  had  permitted 
it,  we  would  have  interviewed  PMs. 

Our  first  interview  agenda  was  too  long  and  difficult; 
the  interviewees  could  not  provide  the  requested 
information.  As  a  result,  we  received  a  limited  number  of 
positive  responses  and  a  great  number  of  negative  responses 
from  our  interviewees .  The  cumbersome  format  resulted  in 
lack  of  participation  by  the  interviewees .  We  corrected 
this  mistake  by  placing  the  easier,  subjective  information 
in  tabular  format  at  the  beginning  of  the  interview.  We 
consolidated  four  pages  of  questions  into  a  table  which  fit 
neatly  onto  one  page.  We  also  moved  the  more  difficult, 
quantitative  historical  questions  to  the  back  of  the 
interview  and  made  them  optional. 

Follow  on  Research.  As  mentioned  in  the  lessons 
learned  section  above,  we  recommend  additional  interviews  be 
conducted  with  Program  Managers,  financial  management 
personnel,  and  TO  users.  The  interviewers  should 
concentrate  on  interviewing  personnel  having  experience  with 
computerized  and  automated  TO  formats .  Our  research 
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revealed  that  TOMAs  are  not  always  as  much  concerned  about 
costs  as  perhaps  a  Program  Manager  would  be.  When  asked  to 
assess  the  relative  importance  of  costs  or  qualitative 
issues  (accuracy,  maintainability,  efficiency, 
effectiveness,  efficiency,  security,  deployability,  or 
timeliness)  TOMAs  responded  overwhelmingly  that  the 
qualitative  issues  are  more  important.  In  other  words, 
beneficial  characteristics  outweigh  costs . 

This  analysis  was  based  on  rough  order  of  magnitude 
estimates.  Limited  historical  data  is  available  for 
computerized  and  automated  TO  formats .  Testing  of  these  TO 
formats  is  taking  place  on  many  different  weapon  systems  in 
many  locations  (for  example  the  F-16  SPO  at  Wright-Patterson 
AFB  and  the  C-130  at  Warner  Robins  ALC) .  The  estimates  and 
assumptions  used  in  this  analysis  can  be  verified  or  refined 
by  including  the  data  from  future  testing.  It  is  important 
that  good  data  be  obtained  from  these  tests,  so  that’  the 
results  can  be  input  into  some  form  of  a  cost  benefit 
analysis . 

The  last  area  of  recommended  future  research  is  the 
computer  simulation  modeling  of  the  entire  TO  system,  or  an 
individual  program  TO  system  (For  excunple  the  F-16)  . 

Computer  simulation  is  a  powerful  tool  which  allows  the 
decision  maker  to  predict  outcomes  without  wasting 
resources.  In  other  words,  simulation  will  provide 
additional  decision  making  data  without  having  to  conduct  a 
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physical  test  on  an  actual  weapon  system,  which  would 
require  expenditures  of  resources . 

Conclusion 

This  research  developed  a  method  for  Program  Managers 
to  implement  when  deciding  which  Technical  Order  format  to 
purchase  for  their  programs.  Costs,  benefits,  and  a  CBA 
procedure  were  identified  by  this  research.  These  factors 
provide  the  PM  with  a  logical  decision  making  tool  that 
utilizes  qualitative  and  quantitative  data.  The  results 
from  this  research  can  be  applied  to  any  program  in  the 
acquisition  or  sustainment  arena. 

This  chapter  lists  the  quantitative  costs  and  the 
qualitative  and  quantitative  benefits  for  all  three  TO 
formats:  paper,  computerized  and  automated.  Once  the  costs 
and  benefits  were  identified,  we  developed  a  decision  making 
tool,  based  on  the  Analytical  Hierarchy  Process,  to  help  the 
decision  maker  choose  the  best  format  for  the  applicable 
program.  The  quantitative  costs  for  a  TO  format  are:  1.) 
Non-recurring  costs:  consulting,  data  acquisition,  hardware, 
software,  telecommunications,  and  training;  and  2.) 
Recurring  costs:  TO  usage,  training,  TO  changes, 
troubleshooting,  communications,  hardware  maintenance, 
software  maintenance,  mishaps  due  to  TO  errors,  mailing 
costs,  telecommunication  costs,  and  AF  TO  management 
manpower.  The  qualitative  benefits  for  a  TO  format  are: 
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accuracy,  maintainability,  efficiency,  effectiveness, 
security,  deployability,  and  timeliness .  A  quantitative 
benefit  of  any  TO  format  is  system  responsiveness. 

Through  the  use  of  a  Life  Cycle  Cost  Analysis,  it  was 
determined  that  the  automated  TO  format  had  the  lowest  life 
cycle  cost.  It  was  also  determined,  by  using  simulation 
modeling,  that  automated  TOs  received  the  highest  rating  for 
system  responsiveness.  As  a  result  of  the  interview 
process,  we  determined  the  paper  format  received  the  highest 
rating  for  QIs .  After  the  data  for  the  three  criteria: 
costs,  system  responsiveness,  and  QIs;  were  input  into  the 
AHP  software,  we  determined  that  the  automated  format  was 
the  best  TO  format  (See  Figure  4  and  Appendix  11) . 

The  costs,  benefits  and  CBA  method  developed  during 
this  research  effort  can  be  applied  to  any  prograun  with 
minimal  effort.  The  only  independent  variables  required  to 
be  changed  when  using  the  CBA  are  quantity  of  TO  pages, 
quantity  of  aircraft,  and  weapon  system  life  cycle. 

However,  additional  accuracy  can  be  obtained  by  further 
refining  assumptions  that  were  used  to  develop  the 
simulation  model,  life  cycle  cost  analysis  and  interview 
process.  The  appendices  contain  the  assumptions  used  to 
develop  the  CBA  as  well  as  the  results  from  the  interviews, 
which  were  generalized  by  including  many  different  System 
Program  Offices. 

The  data  that  was  input  into  the  LCCA,  simulation 
model,  and  AHP  software  contains  estimates  which  may  or  may 
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not  be  accurate.  Once  further  testing  is  accomplished,  the 
results  should  be  used  to  update  and  refine  this  analysis. 
The  important  point  to  remember  is  that  risks  will  be 
incurred  when  implementing  computerized  and  automated 
formats  because  they  make  use  of  new  technology.  The 
objective  of  future  testing  should  be  twofold:  1)  to 
demonstrate  computerization  and  automation  of  TOs  and  2)  to 
provide  good  historical  data  for  input  into  a  decision 
making  technique. 
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Appendix  1:  Glossary  of  Definitions. 


Air  Logistics  Center  (ALC) :  Provides  the  operational 
technical  order  support  required  to  field  the  weapon  system. 
Responsible  for  all  modifications  of  TOs. 

Analytical  Hierarchy  Process  (ABP) :  A  rational  and 
systematic  approach  for  finding  a  solution  to  a  problem. 

The'  method  allows  decision  makers  to  partition  large, 
manageable  problems  into  smaller  parts  that  are  easier  to 
handle.  It  provides  decision  makers  with  the  ability  to 
include  qualitative  and  quantitative  criteria  to  form  a 
rating  for  each  alternative  solution. 

Capital  Budgeting:  A  quantitative  Economic  Analysis 
technique  used  for  capital  investment .  Capital  budgeting 
and  cost  analysis  involves  long  teimi  planning  decisions  for 
investments  using  both  quantitative  and  qualitative  cost 
information . 

Continuous  Acqfuisition  and  Life-cycle  Support  (CALS)  :  A  DOD 
and  Industry  strategy  to  transition  from  a  paper  intensive 
acquisition  and  logistics  database  to  a  highly  automated  and 
integrated  database  for  weapon  systems  of  the  1990 's  and 
beyond . 
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Cost-Benefit  Analysis  (CBA) :  An  analytical  procedure  to 
investigate  and  compare  the  costs  and  benefits  derived  from 
implementing  one  alternative  over  other  alternatives. 

Economic  Analysis:  A  four-step  process:  1)  identification 
of  possible  projects,  2)  data  collection,  3)  determining 
the  best  project,  and  4)  evaluation  of  implemented  projects; 
that  is  used  to  evaluate  changes  to  an  organization  (21:5)  . 

Joint  Continuous  Acquisition  Life-cycle  Support  (JCALS) :  A 
joint  program  (Air  Force,  Army,  Navy,  Marines,  and  the 
Defense  Logistics  Agency  (DLA) )  organized  to  implement  CALS 
throughout  DoD  agencies . 

Life  Cycle  Cost  Analysis:  Analysis  of  all  costs,  using  a 
systems  prospective,  throughout  the  entire  life  cycle  of  a 
program . 

Syston  Program  Office  (SPO) :  The  integrated  AFMC 
organization  responsible  for  cradle-to-grave  military  system 
management  (26:365) . 

Technical  Order  Management  Agency  (TCXIA)  :  A  person  or 
agency  established  to  oversee  the  development  and 
acquisition  of  technical  orders. 
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App«ndix  2:  Glossary  of  Acronyms 


AFSC 

AFTO 

AFTOMS 

AFTOS 

AHP 

ALC 

BET 

CALS 

CBA 

CSRC 

CTOA 

DCF 

DOD 

DODI 

DSSSL 

lETM 

ILSP 

IRR 

lOS 

JCALS 

JUSTIS 

LCC 

LCCA 

LSA 

LSAR 

MCDM 

MIO 

MNS 

NPV 

ORD 

PM 

PMD 

QI 

RC 

ROR 

SQOi 

SPDL 

SPO 

TCTO 

TMMP 

TMPP 

TO 

TOC 

TODCA 


Air  Force  Systems  Command 
Air  Force  Technical  Order 

Air  Force  Technical  Order  Management  System 
Air  Force  Technical  Order  System 
Analytical  Hierarchy  Process 
Air  Logistics  Center 
Break-Even  Time 

Continuous  Acquisition  Life-cycle  Support 

Cost  Benefit  Analysis 

CALS  Shared  Resource  Center 

Central  Technical  Order  Agency 

Discounted  Cash  Flow 

Department  of  Defense 

Department  of  Defense  Instruction 

Document  Style  Semantics  and  Specification 

Language 

Interactive  Electronic  Technical  Manuals 
Integrated  Logistics  Support  Plan 
Internal  Rate  of  Return 
International  Organization  for 
Standardization 

Joint  Continuous  Acquisition  Life-cycle  Support 
Joint  Uniformed  Services  Technical  Information 
System 

Life  Cycle  Cost 
Life  Cycle  Cost  Analysis 
Logistics  Support  Analysis 
Logistics  Support  Analysis  Record 
Multiple  Criteria  Decision  Making 
Management  Integration  Office 
Mission  Need  Statement 
Net  Present  Value 

Organizational  Requirements  Document 

Program  Manager 

Program  Management  Directive 

Qualitative  Issues 

Resource  Centers 

Rate  of  Return 

Standardized  General  Markup  Language 
Standard  Page  Description  Language 
System  Program  Office 
Time  Compliance  Technical  Order 
Technical  Manual  Management  Plan 
Technical  Manual  Publication  Plan 
Technical  Order 
Technical  Order  Center 

Technical  Order  Distribution  Control  Authority 
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TODO 

TOMA 

TQM 


Technical  Order  Distribution  Offices 
Technical  Order  Management  Agency 
Total  Quality  Management 
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J^pp«nciix  3:  Intarviairs  and  Results 

Interview: 

This  interview  gathers  subjective  and  objective 
information  that  will  be  used  to  develop  a  cost  benefit 
analysis  for  implementing  change  to  Air  Force  Technical 
Orders  (TOs) .  Questions  1-6  are  subjective  (there  are  no 
right  or  wrong  answers)  and  7-15  are  objective  (based  on 
historical  information) .  Definitions  of  key  terms  are 
included  at  the  end  of  this  interview. 

1.  What  weapon  aystem(s)  are  you  affiliated  with? 

2.  In  which  area(s)  do  you  have  experience?  (Acquisition, 
Sustainment,  or  Operations) 

4 .  Which  type  of  TOs  do  you  have  experience?  (Paper, 
Computerized,  or  Automated) 

5.  In  column  2,  rank  the  relative  importance  of  each 
characteristic  (a  rank  of  1  means  this  characteristic  is  the 
most  important  characteristic  for  a  TO) . 

6.  Colvimns  3-5  are  three  alternative  forms  of  TOs  (see 
definitions  below) .  Score  each  type  of  TO  columns  3-5)  on  a 
scale  from  1-5.  (1)  meaning  definitely  agree,  (2)  agree,  (3) 
mostly  agree,  (4)  mostly  disagree,  and  (5)  definitely 
disagree  that  the  corresponding  characteristic  is  a  benefit 
for  each  type  of  TO  (i.e.  if  you  definitely  agree  that 
maintainable  is  a  benefit  for  paper  TOs  then  place  a  1  in 
the  row  and  column  corresponding  to  maintainable  and  paper. 


CHARACTERISTIC 

RANK 

PAPER 

COMPUTERIZED 

AUTOMATED 

MAINTAINABLE 

RELIABLE 

SECURITY 

ACCURACY 

EFFICIENT 

EFFECTIVE 

TIMELY 

LOW  COST 

DEPLOYABLE 

FLEXIBLE 

? 

***The  remaining  questions  are  for  TO  acquisition  personnel 
with  financial  /cost  experience*** 
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7 .  What  SPO  or  ALC  are  you  affiliated  with? 

8 .  How  many  TO  pages  were  purchased  for  your  program? 

9 .  Were  the  TOs  paper,  computerized,  or  automated? 

10.  What  was  the  total  cost  for  the  TOs?  Cost  per  page? 

11.  What  cost  benefit  analysis  method  was  used? 

12.  Was  a  cost  benefit  analysis  required? 

13.  If  no  CBA  was  conducted  do  you  think  a  CBA  would  be 
helpful?  Can  you  recommend  a  CBA  method? 

14.  What  were  the  costs  in  the  following  areas: 


AREA 

INITIAL  COSTS 

ANNUAL  COST 

HARDWARE 

SOFTWARE 

TRAINING 

TELECOMMUNICATIONS 

CONSULTATION 

OTHER? 

15.  If  you  purchased  computerized  or  automated  TOs,  are  the 
TOs  currently  in  use?  Are  the  users  satisfied? 

DEFINITIONS 

PAPER  TO:  Technical  Orders  in  paper  form  (Type  A) 
COMPUTERIZED  TO:  Paper  TO  on  computer  file  (Type  B  or  B+) 
AUTOMATED  TO:  Computerized  TOs  within  a  data  base  that  can 
be  accessed  according  to  subject.  (lETM  or  Type  C) 
MAINTAINABLE:  Relative  ease  to  make  changes,  revisions,  and 
corrections . 

RELIABLE:  Probability  the  TO  will  be  available  and  accurate 
when  required. 

SECURITY:  Ability  to  prevent  unauthorized  use  of  the  TO. 
ACCURACY:  Probability  the  TO  has  the  correct  procedure. 
EFFICIENT:  Doing  the  job  the  smartest/quickest  way  possible 
considering  costs . 

EFFECTIVE:  Doing  the  job  the  best  way  possible  without 
considering  costs. 

TIMELY:  Delivery  of  the  TO  at  or  before  the  system 
operation . 

LOW  COST:  The  relative  cost  of  one  system  compared  to  other 
alternatives . 
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DEPLOYABLE:  The  ability  to  take  the  TO  wherever  it  is  needed 
(worldwide) . 

FLEXIBLE:  The  ability  to  incorporate  new  technology  and  or  a 
better  process  with  minimal  disruption  and  costs . 
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IntejTview  Qualitative  Results: 


SPO 

B2-2 

PAPER 

CHARACTERISTIC 

RANK 

PAPER 

MAINTAINABLE 


RELIABLE 


SECURITY 


ACCURACY 


EFFICIENT 


EFFECTIVE 


TIMELY 


LOW  COST 


DEPLOYABLE 


FLEXIBLE 


READABILITY 
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ASC/ALX 

F-15 
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SPO 
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DEPLOYABLE 


FLEXIBLE 
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412 
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SPO 

412LSS 
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J^spendix  4:  Interview  Quantitative  Results 


QUANTITY 

TO 

5700 

QUANTITY 

PAGES 

1, 140, 000 

2124 


2  M  PAGES 


Question 

9 

FORMAT 

PAPER 

PAPER 

PAPER/AUTO 

Question 

10 

COSTS 

$580/PAGE 

$522,666  / 

$246  PAGE 

Question 

CBA 

NO 

NO 

NO 

11 

Question  CBA 
12  REQUIRED 


Question  CBA  WILL 
13  HELP 


Question  COSTS  CAPITAL  EXP 
14 


TRAINING 


COMPUTER 

SW 


TELECOM 


CONVERSION  COST 


YES  (BROOKS) 


$12000  PER  WORK  CENTER 


TEST  ANG  @ 
lA 


Question 

16 

CHANGES 

$3,420,000/YR  $580/PAGE 

$1.5  M  /  YR 

F-15 


QUANTITY 

TO 

490  (DEL)  / 
2200  (ID) 

5000 

QUANTITY 

PAGES 

330,000 

500,000 

C-130 


30,000  - 
35, 000 


Question 

9 

FORMAT 

PAPER 

PAPER 

PAPER 

Question  COSTS 
10- 


Question  CBA 
11 


Question  CBA 
12  REQUIRED 


Question  CBA  WILL 
13  HELP 


$825M  TOT 
($2500/PG) 


$1000  NEW 
$350  CHNG 


$1200  NEW 
$925  CHANGE 


NO  (  CONTR  . 
PROPOSED) 


$300M 


Question  COSTS 
14  CAPITAL 
EXP, 


TRAINING 


COMPUTER 

SW 


TELECOM 


CONVERSION  COST 


NONE 


SYS  230  COST  X  (#) PEOPLE 


NONE 


NONE 


Question  15 

TESTING 

N/A 

TESTING 

Question 

CHANGES 

N/A 

1500-2000 

5000  PAGES 

16 

PAGES  $350- 

450K 
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Appendix  5:  CSRC  Business  Case  Analysis 


Strategic  Questions  to  be  Answered  by  the 
C-17  T.O.  Business  Case  Analysis 

1 .  What  is  the  cost  of  acquiring  and  maintaining  paper  Tech 
Orders? 

2 .  What  is  the  cost  of  acquiring  and  maintaining  digital 
Tech  Orders? 

3.  What  is  the  cost  of  converting  paper  to  digital  today? 

4 .  What  is  the  cost  of  converting  paper  to  digital  when 
JCALS  arrives? 

5.  What  is  the  most  you  would  be  willing  to  pay  McDonnell 
Douglas  for  paper  changes? 

6.  What  is  the  most  you  would  be  willing  to  pay  McDonnell 
Douglas  for  digital  changes? 

Other  Questions? 

1.  How  much  do  C-5  T.O.  changes  cost? 

2 .  What  are  the  fixed  and  variable  costs  for  changing  and 
distributing  paper  T.O.s? 

3.  When  compared  with  the  C-5  program,  how  will  C-17  T.O. 
costs  differ  based  on... 

a.  #  of  aircraft  deployed 

b.  basing  modes 

c.  change  in  maintenance  philosophy 

d.  #  of  personnel  assigned  to  maintenance 

4.  C-5  T.O.  changes  are  prepared  by  the  prime  contractor. 
Can  costs  be  reduced  if  brought  in-house  or  competed? 

a.  ATOS  scenario 

b.  ADMAPS  scenario 
C.  Competed  scenario 

5.  Of  all  T.O.  changes  managed  by  TIRT,  what  percent  are  C- 
5  specific? 

6.  What  are  the  non-financial  benefits  of  digital  T.O.s? 
(e.g.  better  educated  workforce,  timeliness,  accuracy, 
motivations,  etc.). 

7.  What  are  the  indirect  costs  of  TO  change  management? 

8.  What  is  the  cost  of  the  network  infrastructure  needed  to 
support  digital  TOs? 

9.  How  much  does  a  Type  A  document  cost?  (type  B-,  B,  B+, 

C) 
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PAPER  SUMMARY  OUTPUT  REPORT: 

SLAM  II  SUMMARY  REPORT 

SIMULATION  PROJECT  THESIS  BY  PARSONS&BERGIN 

DATE  6/  1/1994  RUN  NUMBER  1  OF  1 


CURRENT  TIME  .1825E+04 

STATISTICAL  ARRAYS  CLEARED  AT  TIME  .OOOOE+00 

‘♦STATISTICS  FOR  VARIABLES  BASED  ON  OBSERVATION** 


MEAN 

STANDARD 

COEFF.  OF 

MINIMUM 

MAXIMUM 

NO.  OF 

VALUE 

DEVIATION 

VARIATION 

VALUE 

VALUE 

OBS 

DEVELOP 

.574E+03 

.898E+02 

.156E+00 

.417E+03 

.761E+03 

28 

MODIFY 

.230E+03 

.796E+02 

.346E+00 

.868E+02 

.441E+03 

81 

“FILE  STATISTICS** 


FILE 

AVERAGE 

STANDARD 

MAXIMUM 

CURRENT 

AVERAGE 

NUMBER 

LABEL/TYPE 

LENGTH 

DEVIATION 

LENGTH 

LENGTH 

WAIT  TIME 

1 

DEV 

QUEUE 

.785 

.715 

3 

1 

35.834 

2 

PRNT 

QUEUE 

.000 

.000 

0 

0 

.000 

3 

DIST 

QUEUE 

.152 

.454 

3 

0 

2.503 

4 

USE 

QUEUE 

206.778 

11.081 

227 

227 

331.317 

5 

FIX 

QUEUE 

.171 

.758 

6 

0 

3.751 

6 

CALENDAR  22.436 

3.089 

29 

21 

14.587 

“REGULAR  ACTIVITY  STATISTICS** 

ACTIVITY 

AVERAGE 

STANDARD 

MAXIMUM  CURRENT 

ENTITY 

INDEX/LABEL 

UTILIZATION 

DEVIATION 

UTIL 

UTIL 

COUNT 

1 

.0000 

.0000 

1 

0 

40 

* ‘SERVICE  ACTIVITY  STATISTICS** 


ACT 

ACT 

LABEL  OR 

SER 

AVERAGE 

STD 

CUR 

AVERAGE 

MAX  IDL 

MAX  BSY 

ENT 

NUM 

START  NODE 

CAP 

UTIL 

DEV 

UTIL 

BLOCK 

TME/SER 

TME/SER 

CNT 

2 

DEV 

QUEUE 

8 

7.289 

1.74 

8 

.00 

8.00 

8.00 

31 

4 

PRNT 

QUEUE 

20 

9.898 

2.92 

9 

.00 

20.00 

16.00 

111 

5 

DIST 

QUEUE 

2 

.891 

.82 

2 

.00 

2.00 

2.00 

109 

6 

USE 

QUEUE 

1 

.088 

.28 

1 

.00 

116.00 

4.00 

80 

7 

USE 

QUEUE 

1 

,912 

.28 

0 

.00 

4.00 

116.00 

832 

8 

FIX 

QUEUE 

7 

2.375 

1.71 

1 

.00 

7.00 

7.00 

89 
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COMPUTERIZED  SUMMARY  OUTPUT  REPORT: 


SLAM  II  SUMMARY  REPORT 


SIMULATION  PROJECT  THESIS  BY  PARSONS&BERGIN 

DATE  6/  1/1994  RUN  NUMBER  1  OF  1 

CURRENT  TIME  .1825E+04 

STATISTICAL  ARRAYS  CLEARED  AT  TIME  .OOOOE+00 

‘♦STATISTICS  FOR  VARIABLES  BASED  ON'  OBSERVATION** 


MEAN 

STANDARD 

COEFF.  OF 

MINIMUM 

MAXIMUM 

NO.  OF 

VALUE 

DEVIATION 

VARIATION 

VALUE 

VALUE 

OBS 

DEVELOP 

.444E+03 

.342E+02 

.771E-01 

.333E+03 

.495E+03 

30 

MODIFY 

.749E+02 

.160E+02 

.214E+00 

.462E+02 

.131E+03 

69 

“FILE  STATISTICS** 


FILE 

AVERAGE 

STANDARD 

MAXIMUM 

CURRENT 

AVERAGE 

NUMBER 

LABEL/TYPE 

LENGTH 

DEVIATION 

LENGTH 

LENGTH 

WAIT  TIME 

1 

DEV 

QUEUE 

1.013 

.837 

3 

1 

46.223 

2 

PRNT 

QUEUE 

.000 

.000 

0 

0 

.000 

3 

DIST 

QUEUE 

.000 

.000 

0 

0 

.000 

4 

USE 

QUEUE 

216.794 

10.913 

235 

235 

344.942 

5 

FIX 

QUEUE 

.145 

.712 

6 

0 

4.067 

6 

CALENDAR  12.371 

1.833 

19 

13 

8.491 

* 

‘REGULAR  ACTIVITY  STATISTICS** 

ACTIVITY 

AVERAGE 

STANDARD 

MAXIMUM  CURRENT 

ENTITY 

INDEX/LABEL 

UTILIZATION 

DEVIATION 

UTIL 

UTIL 

COUNT 

1 

.0000 

.0000 

1 

0 

40 

‘‘SERVICE  ACTIVITY  STATISTICS** 


ACT 

ACT  LABEL  OR 

SER 

AVERAGE 

STD 

CUR 

AVERAGE 

MAX  IDL 

MAX  BSY 

ENT 

NUM 

START  NODE 

CAP 

UTIL 

DEV 

UTIL 

BLOCK 

TME/SER 

TME/SER 

CNT 

2 

DEV  QUEUE 

8 

7.292 

1.74 

8 

.00 

8.00 

8.00 

31 

5 

DIST  QUEUE 

2 

.056 

.23 

1 

.00 

2.00 

2.00 

101 

4 

PRNT  QUEUE 

20 

1.118 

1.19 

2 

.00 

20.00 

7.00 

99 

6 

USE  QUEUE 

1 

.068 

.25 

1 

.00 

118.00 

4.00 

62 

7 

USE  QUEUE 

1 

.932 

.25 

0 

.00 

4.00 

118.00 

850 

8 

FIX  QUEUE 

7 

1.922 

1.70 

1 

.00 

7.00 

7.00 

71 

100 


AUTOMATED  SUMMARY  OUTPUT  REPORT: 

SLAM  II  SUMMARY  REPORT 


SIMULATION  PROJECT  THESIS  BY  PARSONS&BERGIN 

DATE  6/  1/1994  RUN  NUMBER  1  OF  1 


CURRENT  TIME  .1825E+04 

STATISTICAL  ARRAYS  CLEARED  AT  TIME  .OOOOE+00 


**STATISTICS  FOR  VARIABLES  BASED  ON  OBSERVATION** 


MEAN 

STANDARD 

COEFF.  OF 

MINIMUM 

MAXIMUM 

NO.  OF 

VALUE 

DEVIATION 

VARIATION 

VALUE 

VALUE 

OBS 

DEVELOP 

.356E+03 

.469E+02 

.132E+00 

.251E+03 

.450E+03 

32 

MODIFY 

.456E+02 

.122E+02 

.267E+00 

.259E+02 

.809E+02 

47 

**FILE  STATISTICS** 


FILE 

AVERAGE 

STANDARD 

MAXIMUM 

CURRENT 

AVERAGE 

NUMBER 

LABEL/TYPE 

LENGTH 

DEVIATION 

LENGTH 

LENGTH 

WAIT  TIME 

1 

DEV 

QUEUE 

.086 

.280 

1 

0 

3.927 

2 

PRNT 

QUEUE 

.000 

.000 

0 

0 

.000 

3 

DIST 

QUEUE 

.000 

.000 

0 

0 

.000 

4 

USE 

QUEUE 

220,049 

11.055 

240 

240 

348.603 

5 

FIX 

QUEUE 

.058 

.406 

3 

0 

2.630 

6 

CALENDAR 

10.130 

1.503 

15 

9 

7.536 

* 

*REGULAR  ACTIVITY  STATISTICS** 

ACTIVITY 

AVERAGE 

STANDARD 

MAXIMUM 

[  CURRENT 

ENTITY 

INDEX/LABEL 

UTILIZATION 

DEVIATION 

UTIL 

UTIL 

COUNT 

1 

.0000 

.0000 

1 

0 

40 

**SERVICE  ACTIVITY  STATISTICS** 


ACT 

ACT  LABEL  OR 

SER 

AVERAGE 

STD 

CUR 

AVERAGE 

MAX  IDL 

MAX  BSY 

ENT 

NUM 

START  NODE 

CAP 

UTIL 

DEV 

UTIL 

BLOCK 

TME/SER 

TME/SER 

CNT 

2 

DEV.  QUEUE 

8 

7.012 

1,69 

8 

.00 

8.00 

8.00 

32 

4 

PRNT  QUEUE 

1 

.000 

.00 

0 

.00 

137.71 

.00 

79 

5 

DIST  QUEUE 

2 

.043 

.21 

0 

.00 

2.00 

2.00 

79 

6 

USE  QUEUE 

1 

.041 

.20 

1 

.00 

136.00 

2.00 

37 

7 

USE  QUEUE 

1 

.959 

.20 

0 

.00 

2.00 

136.00 

875 

8 

FIX  QUEUE 

7 

1.091 

1.40 

0 

.00 

7.00 

7.00 

47 
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A{>p«ndix  7 :  SLAM  II  Assimptions 


The  simulation  models  contain  rough  order  magnitude 
(ROM)  estimates  for  all  time  distributions.  Basically  the 
paper  model  simulates  the  current  paper  (G022)  system. 
Computerized  and  automated  systems  have  very  similar  develop 
and  review  processes  and  the  main  differences  between  them 
and  the  paper  model  is  the  print  process.  The  computerized 
model  assumes  the  TOs  are  printed  at  the  base  level  and  the 
automated  model  assiimes  the  TO  is  never  printed. 

Time  distributions  for  develop,  print,  distribute,  use, 
and  fix  TO  processes  were  estimated  by  the  research  team, 
based  on  the  literature  review  and  interviews,  and  inputted 
into  the  model.  The  estimates  are  as  follows: 

All  Three  Models  Assume: 

1.  There  are  approximately  200,000  TOs  in  use  in  the 
USAF  inventory  and  another  10,000  TOs  at  the  ALC  to  be 
fixed. 

2.  Eight  thousand  TOs  are  created  annually. 

3.  All  models  follow  the  same  chronological  processes: 
develop,  print,  distribute,  use,  and  fix  except  the 
paper  model  includes  a  print  process  prior  to 
distribution  and  the  computerized  requires  printing 
after  the  distribution  process.  The  automated  model 
does  not  require  the  print  process .  If  a  current  TO 
requires  a  change,  it  does  not  require  the  develop 
process  because  the  develop  process  is  for  new  TOs. 

The  Paper  Model  Assumes: 

1.  Develop  TO:  follows  a  normal  distribution  with  a 
mean  of  385  days  and  standard  deviation  (SD)  of  40  days 
and  contractor's  capacity  is  unlimited. 

2.  Print  TO:  follows  a  triangular  distribution  with  a 
mean  of  90  days,  a  low  of  ten  and  a  high  of  365  days. 
Capacity  is  20,000  TOs. 

3.  Distribute  TO:  follows  a  normal  distribution  with  a 
mean  of  15  days,  SD  of  5  days  and  capacity  of  2,000 
TOs. 

4.  Use  TO:  Eight  %  of  TOs  require  changes  annually. 

5.  Fix  TO:  follows  a  log  normal  distribution  with  a 
mean  of  50  days,  SD  of  10  days,  and  capacity  of  7,000 
TOs. 
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The  Coo^uterized  Model  Assumes : 

1.  Develop  TO:  follows  a  normal  distribution  with  a 
mean  of  385  days  and  standard  deviation  (SD)  of  40  days 
and  contractor's  capacity  is  unlimited. 

2.  Print  TO:  follows  a  normal  distribution  with  a  mean 
of  20  days,  SD  of  5  days 

3.  Distribute  TO:  requires  just  one  day  and  has  no 
capacity  limits  for  TO  quantities. 

4.  Use  TO:  Six  %  of  TOs  require  changes  annually. 

5.  Fix  TO:  follows  a  log  normal  distribution  with  a 
mean  of  50  days,  SD  of  10  days,  and  capacity  of  7,000 
TOs . 

The  Automated  Model  Assumes: 

1.  Develop  TO:  follows  a  normal  distribution  with  a 
mean  of  360  days  and  standard  deviation  (SD)  of  40  days 
and  contractor's  capacity  is  unlimited. 

2.  Print  TO:  not  required. 

3.  Distribute  TO:  requires  just  one  day  and  has  no 
capacity  limits  for  TO  quantities. 

4.  Use  TO:  Three  %  of  TOs  require  changes  annually. 

5.  Fix  TO:  follows  a  log  normal  distribution  with  a 
mean  of  40  days,  SD  of  10  days,  and  capacity  of  7,000 
TOs . 
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Appendix  8:  Assumptions  for  Life  Cycle  Cost 
Analysis  (LCCA)  Formulation 


KEY  ASSUMPTIONS:  Cost  estimates  for  paper  and  computer  TO 
formats  were  obtained  from  the  literature  review  and 
interview  processes.  Additionally,  this  analysis  contains 
rough  order  magnitude  (ROM)  cost  estimates  developed  by  the 
research  team  when  literature  review  and  interview  data  did 
not  provide  the  estimates . 

Cost  estimates  for  automated  TO  format  was  obtained 
from  the  Armstrong  Laboratory  cost  benefit  study  conducted 
on  the  F-16  aircraft.  This  study  converted  the  Armstrong 
Laboratory's  ROM  estimates  into  unit  costs  for  one  aircraft 
and  TO  page. 

Coats  are  divided  into  two  categories,  non  recurring 
and  recurring,  for  this  LCCA. 

NON  RECURRING  COSTS 

CONSULT:  Consulting  includes  systems  engineering  and  program 
management  efforts  required  to  develop  a  particular  TO 
format . 


PAPER: 


COMPUTERIZED; 


AUTOMATED ; 


Paper  format  already  developed:  $0  cost. 

ROM  estimate  $5  /  page  (pg.) . 

$12.8  Million  M)/575,000  pgs  =  $22.3/pg 
(29:32) . 


DATA:  The  technical  order  final  product  to  be  delivered 

to  the  end  user. 

FORMAT  ASSUMPTIONS  /  RATIONALE 

PAPER;  Based  on  interviews  $1,000  /  pg. 


COMPUTERIZED : 


AUTOMATED : 


ROM  estimate  $1200  /  pg. 

ROM  $400  /  pg  due  to  increased  efficiency 
from  Air  Force  Authoring  System. 
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HARDWARE:  Hardware  required  to  support  the  alternative  TO. 


FORMAT  -ASSUMPTIONS  /  RATIONALE 

PAPER:  Basecase  assumed  $0  to  support  paper  format 

other  format  costs  are  relative  to  paper. 

COMPUTERIZED:  ROM  estimate  $25,000  /  aircraft  (acft) . 

AUTOMATED:  $97.7  M  /  1700  aircraft  -  $57,471/acft 

(29:32)  . 

SOFTWARE:  Software  required  to  support  TO  format. 

format  assumptions  /  RATIONALE 

PAPER:  Basecase  assumes  $0  to  support  paper. 

COMPUTERIZED:  ROM  estimate  $2  /  pg. 

AUTOMATED:  $6.7  M  /  575,000  pages  -  $11.7  /  pg 

(29:32). 

TELECOMMUNICATIONS:  Telecommunications  will  be  implemented 

regardless  of  which  format  is  chosen  so  this  is  not  a 
relevant  cost  for  this  life  cycle  analysis. 

TRAINING:  Initial  training  required  to  support  TO 

format . 


FORMAT 


ASSUMPTIONS  /  RATIONALE 


PAPER:  Basecase  assumes  $0  to  support  paper. 

COMPUTERIZED:  ROM  estimate  1/4  of  automated  training  costs. 

AUTOMATED:  $2.5  M  /  1700  acft  -  $1470  /  acft  (29:32). 

DATA  Q3HVERSION:  Cost  to  convert  data  from  paper  into 
another  format. 


FORMAT  ASSUMPTIONS  /  RATIONALE 

PAPER:  Basecase  assumes  $0  to  support  paper  (no 

conversion  necessary) . 

COMPUTERIZED:  $15  /  pg  estimate  from  interview  (32) 

AUTOMATED:  $54.6  M  /  575,000  pgs  or  $95  /  pg  (29:32). 
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RECURRING  /  ANNUAL  COSTS 


TO  USAGE:  Costs  due  to  manual  posting  of  TO  changes,  TO 
research  time,  weight  required  for  deployment,  and  effort 
required  to  submit  AFTO  22  change  requests . 


ASSUMPTIONS  /  RATIONALE 

PAPER:  $73.4  M  /  1700  acft  =  $43,177  /  acft 

(29:33) . 


COMPUTERIZED:  Same  as  paper  format  or  $43,177  /  acft. 

AUTOMATED:  $8.4  M  /  1700  acft  =  $4,941  /  acft 

(29:33)  . 


TRAINING:  Recurring  training  required  to  support  TO 

format . 

ASSUMPTIONS  /  RATIONALE 
PAPER:  Basecase  assumes  $0. 

COMPUTERIZED:  Additional  follow  on  computer  training 
negligible  and  to  be  accomplished  on  the  job. 

AUTOMATED:  Additional  follow  on  computer  training 

negligible  and  to  be  accomplished  on  the  job. 

TO  CHANGES:  Cost  to  correct  deficiencies  within  TOs . 

ASSUMPTIONS  /  RATIONALE 

Eight  percent  of  TOs  require  annual  changes 
at  $500  /  pg  (l:vii). 

ROM  estimate  6  percent  of  TOs  require 
annual  changes  at  $300  /  pg. 

ROM  estimate  3  percent  of  TOs  require 
annual  changes  at  $300  /  pg. 


EQBMRT 

PAPER: 

COMPUTERIZED : 

AUTOMATED : 
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DIAGNOSTICS:  Costs  due  to  maintenance  troubleshooting 
time,  retest  OKs,  repair  time. 

FORMAT  ASSUMPTIONS  /  RATIONALE 

PAPER:  $146.7  M  /  1700  acft  =  $86,309  /  acf t . 


COMPUTERIZED:  Same  as  paper  format. 

AUTOMATED:  $100  M  /  1700  acft  -  $58,824  /  acft. 

COMMUNICATION:  Costs  due  to  information  system 

interactions  and  part  ordering. 

FORMAT  ASSUMPTIONS  /  RATIONALE 

PAPER:  $8.9  M  /  1700  acft  =  $5,250  /  acft 

(29:33)  . 


COMPUTERIZED:  Same  as  paper  format. 

AUTOMATED:  $3.9  M  /  1700  acft  =  $2,294  /  acft 

(29:33)  . 

HARDWARE  MAINTENANCE:  Cost  to  maintain  hardware. 

assumptions . /  .rationale 

PAPER:  Basecase  cost  =  $0. 


COMPUTERIZED:  ROM  estimate  $200  /  acft. 

AUTOMATED:  $5.4  M  /  1700  acft  =*  $3,177  /  acft 

(29:33)  . 

SOFTWARE  MAINTENANCE:  Cost  to  maintain  software. 


FORMAT  ASSUMPTIONS  /  RATIONALE 

PAPER:  Basecase  cost  =  $0. 


COMPUTERIZED:  ROM  estimate  is  $200  /  acft. 
AUTOMATED:  $8.5  M  /  1700  acft  -  $5,000  /  acft. 
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MISHAPS:  Costs  incurred  from  class  I  mishaps  that  indicated 
TOs  as  contributing  factor. 

FORMAT  ASSUMPTIONS  /.RATIQNALS 


PAPER: 


$8.5  M  /  15,000  total  acft  in  AF  inventory  = 
$567  /  acft  (1:1-6) . 


COMPUTERIZED:  Same  as  paper  format. 


AUTOMATED:  ROM  estimate  90  percent  reduction  in 

mishaps  due  to  TO  errors  or  $57  /  acft. 


MAIL:  Costs  due  to  mailing  changes. 


FORMAT 

PAPER: 


COMPUTERIZED : 
AUTOMATED : 


ASSUMPTIONS  /  RATIONALE 

Eight  percent  of  TOs  change  annually  at  $10 
/  pg  ROM  mailing  costs . 

No  mailing  costs  for  computerized  format . 

No  mailing  costs  for  automated  format. 


TELBCOMMDNICATIONS :  Costs  for  sending  TO  changes  in 
electronic  format . 


FORMAT 

PAPER: 

COMPUTERIZED ; 

AUTOMATED : 

AF  MANPOWER: 
FORMAT 

PAPER: 

COMPUTERIZED ; 
AUTOMATED : 


ASSUME!  IQN.S-  Z_RAT10NALE 
No  costs  for  paper  format . 

Six  percent  of  TOs  require  changes  and  ROM 
cost  $3  /  pg  to  transmit . 

Three  percent  of  TOs  require  changes  and 
ROM  cost  $3  /  pg. 

Overhead  costs  for  management  structure . 

ASSUMETI0NS-./-BAT10NALE 
ROM  estimate  $500  /  pg. 

ROM  estimate  $250  /  pg. 

ROM  estimate  $75  /  pg. 
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Appendix  9:  Life  Cycle  Cost  Analysis  Formulas 


TO  LIFE  CYCLE  COST  ANALYSIS 
FORMULAS 

ORGANIZATION: 

F-16  SPO 

ACFT 

LIFE  CYC 

TO 

(QTY) 

(#  YRS) 

(#PGS) 

1,700 

10 

575, 000 

NON  RECURRING  COSTS 

CONSULT 

DATA 

PAPER 

0 

1000*$C$7 

COMPUT 

5*C7 

1200*$C$7 

AUTO 

22.3*C7 

400*$C$7 

RECURRING  COSTS 

TO  USAGE 

TRNG 

PAPER 

43177*A7 

mmEmm 

COMPUT 

43177*A7 

0 

AUTO 

4941*A7 

0 

RECURRING  COSTS 

MISHAPS 

MAIL 

PAPER 

567*A7 

0 .08*07*10 

COMPUT 

567*A7 

0 

AUTO 

57*A7 

.  0 
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NONRECURRING 

COSTS 

HDWRE 

SOFTWR 

TELECOM 

TRNG 

DATA  C 

1000*A7 

0.5*C7 

0 

0 

0 

25000*A7 

2*C7 

0 

350*A7 

15*C7 

57471*A7 

12*C7 

0 

1470*A7 

95*C7 

RECURRING 

ANNUAL 

COSTS 

TO  CHGS 

TRBL  SHT 

COMMUN 

HDWMX 

SFW  MX 

0.08*C7*500 

86309*A7 

0 

0 

0.06*C7*300 

5250*A7 

200*A7 

0 

0.03*C7*300 

58824*A7 

2294*A7 

3177*A7 

5000*A7 

RECURRING 

(CONT.) 

TELECOM, 

AF  MANPOWER 

0 

0.06*C7*3 

250*C7 

0.03*C7*3 

75*C7 

• 
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YEAR 

PAPER  COST  FORMULAS 

1 

B12+C12+D12+E12+F12+G12+B19+C19+D19+E19+F19 
+G1 9+H19+B2  6+C2  6+D2  6+E2  6 

2 

$B$19+$C$19+$D$19+$E$19+$F$19+$G$19+$H$19+$ 

B$26+$C$26+$D$26+$E$26 

3 

$B$19+$C$19+$D$19+$E$19+$F$19+$G$19+$H$19+$ 

B$26+$C$26+$D$26+$E$26 

4 

$B$19+$C$19+$D$19+$E$19+$F$19+$G$19+$H$19+$ 

B$26+$C$26+$D$26+$E$26 

5 

$B$19+$C$19+$D$19+$E$19+$F$19+$G$19+$H$19+$ 

B$26+$C$26+$D$26+$E$26 

6 

$B$19+$C$19+$D$19+$E$19+$F$19+$G$19+$H$19+$ 

B$26+$C$26+$D$26+$E$26 

7 

$B$19+$C$19+$D$19+$E$19+$F$19+$G$19+$H$19+$ 

B$26+$C$26+$D$26+$E$26 

8 

$B$19+$C$19+$D$19+$E$19+$F$19+$G$19+$H$19+$ 

B$26+$C$26+$D$26+$E$26 

9 

$B$19+$C$19+$D$19+$E$19+$F$19+$G$19+$H$19+$ 

B$26+$C$26+$D$26+$E$26 

10 

$B$19+$C$19+$D$19+$E$19+$F$19+$G$19+$H$19+$ 

B$26+$C$26+$D$26+$E$26 

TOTAL  (NO  TIME 
VALUE  OF  $) 

SUM(B39:B48) 

TOTAL  (NPV 
0.07) 

NPV{0.07,B39:B48) 
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YEAR 


COMPUTERIZED  COST  FORMULAS 


B13+C13+D13+E13+F13+G13+B20+C20+D20+E20+F20 
+G2  0+H2  0 +B2  7  +C2  7  +D2  7  +E2  7 


$B$20+$C$20+$D$20+$E$20+$F$20+$G$20+$H$20+$ 

B$27+$C$27+$D$27+$E$27 


$B$20+$C$20+$D$20+$E$20+$F$20+$G$20+$H$20+$ 

B$27+$C$27+$D$27+$E$27 


$B$20+$C$20+$D$20+$E$20+$F$20+$G$20+$H$20+$ 

B$27+$C$27+$D$27+$E$27 


$B$20+$C$20+$D$20+$E$20+$F$20+$G$20+$H$20+$ 

B$27+$C$27+$D$27+$E$27 


$B$20+$C$20+$D$20+$E$20+$F$20+$G$20+$H$20+$ 

B$27+$C$27+$D$27+$E$27 


$B$20+$C$20+$D$20+$E$20+$F$20+$G$20+$H$20+$ 

B$27+$C$27+$D$27+$E$27 


$B$20+$C$20+$D$20+$E$20+$F$20+$G$20+$H$20+$ 

B$27+$C$27+$D$27+$E$27 


$B$20+$C$20+$D$20+$E$20+$F$20+$G$20+$H$20+$ 

B$27+$C$27+$D$27+$E$27 


$B$20+$C$20+$D$20+$E$20+$F$20+$G$20+$H$20+$ 

B$27+$C$27+$D$27+$E$27 


TOTAL  (NO  TIME 
VALUE  OF  $) 

SUM(C39:C48) 

TOTAL  (NPV 
0.07) 

NPV(0.07,C39:C48) 
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YEAR 

AUTOMATED  COST  FORMULAS 

_ 

1 

B14+C14+D14+E14+F14+G14+B21+C21+D21+E21+F21 
+G2 1+H21+B28+C28+D28+E28 

2 

$B$21+$C$21+$D$21+$E$21+$F$21+$G$21+$H$21+$ 

B$28+$C28+$D$28+$E$28 

3 

$B$21+$C$21+$D$21+$E$21+$F$21+$G$21+$H$21+$ 

B$28+$C29+$D$28+$E$28 

4 

$B$21+$C$21+$D$21+$E$21+$F$21+$G$21+$H$21+$ 

B$28+$C30+$D$28+$E$28 

5 

$B$21+$C$21+$D$21+$E$21+$F$21+$G$21+$H$21+$ 

B$28+$C30+$D$28+$E$28 

6 

$B$21+$C$21+$D$2l+$E$21+$F$21+$G$21+$H$21+$ 

B$28+$C32+$D$28+$E$28 

7 

$B$21+$C$21+$D$21+$E$21+$F$21+$G$21+$H$21+$ 

B$28+$C33+$D$28+$E$28 

8 

$B$21+$C$21+$D$21+$E$21+$F$21+$G$21+$H$21+$ 

B$28+$C34+$D$28+$E$28 

9 

$B$21+$C$21+$D$2l+$E$21+$F$21+$G$21+$H$21+$ 

B$28+$C35+$D$28+$E$28 

10 

$B$21+$C$21+$D$21+$E$21+$F$21+$G$21+$H$21+$ 

B$28+$C36+$D$28+$E$28 

TOTAL  (NO  TIME 
VALUE  OF  $) 

SUM(D39:D48) 

TOTAL  (NPV 
@.07) 

NPV(0.07,D39:D48) 

4 
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Appendix  10:  Life  Cycle  Cost  Analysis  Output 


TO  LIFE  CYCLE  COST  ANAL 


ACFT 

LIFE  CYC 

TO 

(QTY) 

(#  YRS) 

(#PGS) 

1,700 

10 

575,000 

PAPER 


COMPUT 


AUTO 


CONSULT 


2,875,000 


12,822,500 


DATA 


575,000,000 


690,000,000 


230, 000,000 


TO  USAGE 


PAPER 

73,400, 900 

COMPUT 

73,400, 900 

AUTO 

8,399,700 

TRNG 


0 


PAPER 

963,900 

COMPUT 

963, 900 

AUTO 

96,900 

SOFTWR 


287,500 


1,150,000 


6, 900, 000 


(COST/ACFT  YR) 


TRBL  SHT 


146,725,300 


146,725,300 


100,000,800 


AF  MANPOWER 


287,500,000 


143,750,000 


43,125,000 


MAIL 


460,000 


0 


(COST/ACFT) 


TELECOM 


0 


0 


0 


COMMUN 


8, 925, 000 


8,925,000 


3,899,800 


NONRECURRING  COSTS 


HDWRE 


1,700,000 


42,500,000 


97, 700, 700 


RECURRING  /  ANNUAL  COST: 


TO  CHGS 


23, 000, 000 


10,350,000 


5,175,000 


RECURRING  (COST/ACFT  YR) 


TELECOM, 


0 


103,500 


51,750 


TRNG 


0 


595, 000 


2,499,000 


HDWMX 


0 


340,000 


5,400,900 


IDATA  C 


0 


8, 625, 000 


16, 100, 000 


SFW  MX 


0 


0 
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YEAR 


YEAR 

PAPER 

1 

1, 117, 962,  600 

2 

540,975,100 

3 

540,975,100 

4 

540, 975, 100 

5 

540, 975, 100 

6 

540,975,100 

7 

540, 975, 100 

8 

540,975,100 

9 

540, 975, 100 

10 

540, 975, 100 

TOTAL  (NO  TIME  VALUE  OF  $) 

5,986,738,500 

TOTAL  (NPV  @.07) 

4,338,823,381 

YEAR 

COMPUTERIZED 

1 

1, 121, 678, 600 

2 

384,558,600 

3 

384,558, 600 

4 

384,558, 600 

5 

384,558,600 

6 

384,558,600 

7 

384,558, 600 

8 

384,558,600 

9 

384,558,600 

10 

384,558,600 

TOTAL  (NO  TIME  VALUE  OF  $) 

4,582,706, 000 

TOTAL  (NPV  @.07) 

3,389,875,881 

YEAR 

AUTOMATED 

1 

524,572,050 

2 

174, 649, 850 

3 

174,649,850 

4 

174, 649, 850 

5 

174, 649,850 

6 

174,649,850 

7 

174,649,850 

8 

174, 649, 850 

9 

174, 649,850 

10 

174,649,850 

TOTAL  (NO  TIME  VALUE  OF  $) 

2,096,420,700 

TOTAL  (NPV  @.07) 

1,553,697,556 

Appendix  11:  Analytical  Hierarchy  Fomulation 
Charts  and  Performance  Report 


Dr.  David  S.  Christensen  -  Evaluation  copy 


GOAL 

L  1.000 
G  1.000 


COST 

L  0.500 
G  0.500 


—PAPER 
L  0.250 
G  0.125 
—COMP 
L  0.250 
G  0.125 
— ADTO 
L  0.500 
G  0.250 


SYS  RESP 

L  0.300 
G  0.300 


—MODIFY 
L  0.600 
G  0.180 
—DEVELOP 
L  0.400 
G  0.120 


QIS 

L  0.200 
6  0.200 


—ACCURACY 
L  0.358 
G  0.072 
— EFFIN 
L  0.240 
G  0.048 
— EFFEC 
L  0.155 
G  0.031' 
-TIMELY 
L  0.104 
G  0.021 
-MAINT 
L  0.068 
G  0.014 
—DEPLOY 
L  0.044 
G  0.009 
-SECURITY 
L  0.031 
G  0.006 
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Glossary  of  Terms  Used  In  Model 
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COST 


L  0.500 
G  0.500 


SYS  RESP 


L  0.300 
G  0.300 


MODIFY 

L  0.600 
G  0.180 


—AUTO 
L  0.545 
G  0.098 
—COMP 
L  0.343 
G  0.062 
—PAPER 
L  0.112 
G  0.020 


DEVELOP 

L  0.400 
G  0.120 


—COMP 
L  0.331 
G  0.040 
—PAPER 
L  0.256 
G  0.031 
—AUTO 
L  0.413 
G  0.050 


MODIFY 
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MM 
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- 

- 

HAIIIT 

L  0.068 
G  0.014 

PAPER 

T 
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AOTO 

L  0.200 

1 

L  0.400 

L  0.400 
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1 
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